Introduction

By default, Docker container logs only show stdout and stderr standard streams, which are cleared when the container is destroyed. However, when non-interactive processes, like a database or web server are run, logs pertaining to these processes are generated. These logs can be collected using Docker logging drivers and saved to a file. If the design of a container is good enough, it can provide valuable logs to be examined. This blog post addresses collecting Docker container logs and using Wazuh to analyze them.

Example setup

  • Wazuh manager: It will take care of processing, through decoders and rules, the container data collected by the agent.
  • Linux/Unix endpoint: This will host the Docker containers that will be created and monitored. Wazuh agent, Docker, and Rsyslog are assumed to be installed on this system.

Configure Docker to use the Syslog logging driver (Linux/Unix endpoint)

By default, when a container is created by the docker engine, the JSON File logging driver is used. Logs captured this way can be found in /var/lib/docker/containers/[container-id]/[container-id]-json.log. The Wazuh agent can collect logs directly from this location but the logs do not provide enough information about which container produced it. This might be a problem when we have a large number of containers. Rsyslog is used to improve the container logs by tagging them with their respective container names and grouping them in a directory.

The JSON File logging driver doesn’t allow the log file location to be changed, so we will use the Syslog logging driver instead.

When creating a container, the --log-driver tag can be specified to set the log driver which will be used. A more global approach is to use a configuration that sets the logging driver for all containers created.

To set the global logging driver, create or edit /etc/docker/daemon.json. This way all containers created henceforth will use the Syslog driver:

{
  "log-driver": "syslog",
  "log-opts": {
    "syslog-address": "unixgram:///dev/log",
    "tag": "docker/{{.Name}}"
  }
}

The configuration above specifies:

  • The type of log driver used, which in this case is Syslog.
  • That log messages will be channeled to unixgram:///dev/log, then the Rsyslog daemon pulls data from that socket.
  • That log messages will be tagged with docker/{{.Name}}, where {{.Name}} replaces the container name.

Restart the Docker service to apply changes:

systemctl restart docker

Configure Rsyslog (Linux/Unix endpoint)

We need to configure Rsyslog to create and tag new logs from containers in the /var/log/docker folder.

Let’s create a configuration file called wazuh-docker.conf in the /etc/rsyslog.d/ directory:

$FileCreateMode 0644
$template DockerDaemonLogFileName,"/var/log/docker/docker.log"
$template DockerContainerLogFileName,"/var/log/docker/%SYSLOGTAG:R,ERE,1,FIELD:docker/(.*)\[--end:secpath-replace%.log"
if $programname == 'dockerd' then {
  ?DockerDaemonLogFileName
  stop
}
if $programname == 'containerd' then {
  ?DockerDaemonLogFileName
  stop
}
if $programname == 'docker' then {
  if $syslogtag contains 'docker/' then {
  ?DockerContainerLogFileName
  stop
  }
}
$FileCreateMode 0600

The configuration above specifies:

  • The files created are in 0644 mode, meaning they will be readable by all the user groups, but writable by the user only.
  • That any log from dockerd, containerd, and docker runtime will be written to /var/log/docker/docker.log.
  • SYSLOGTAG:R,ERE,1,FIELD:docker/(.*)\[--end:secpath-replace% is equivalent to the {{.Name}} tag in the Docker daemon configuration. Therefore new logs will be located in /var/log/docker/[new container name].log.

Restart the Rsyslog service to apply changes:

systemctl restart rsyslog

New logs should start appearing in the /var/log/docker directory when containers are created.

Testing the configuration with a web server container

Linux/Unix endpoint

To configure log collection locally on the agent, add the configuration below inside the <ossec_config> tag of the /var/ossec/etc/ossec.conf file. A wild card is used, this way the Wazuh agent will read every container log in that directory:

<localfile>
  <log_format>syslog</log_format>
  <location>/var/log/docker/*</location>
</localfile>

Restart the Wazuh agent for changes to apply:

systemctl restart wazuh-agent

As mentioned earlier, non-interactive applications produce additional logs, depending on the design of the container. For example, a web server container will generate web logs. So we expect to see HTTP web requests like “GET” in the logs.

Let’s set up a simple web server that shows its hostname, IP address, and other information:

  • Create a container using the nginxdemos/hello container image. Setting the container name is important for Rsyslog to tag the log:
docker run -P -d --hostname webserver --name webserver nginxdemos/hello
  • Check that there is a new log file:
ls /var/log/docker/webserver.log

Wazuh manager

Configure the manager to store events collected by the agent to /var/ossec/logs/archives/archives.json file. These logs will be used to create rules and decoders. In the /var/ossec/etc/ossec.conf file, change <logall_json> to “yes”. Later in this blog post when rules are being triggered, we change <logall_json> back to “no”. This way large volumes of logs are not stored on the manager side:

<logall_json>yes</logall_json>

Restart the Wazuh manager for changes to apply:

systemctl restart wazuh-manager

Testing the configurations

On the Linux/Unix endpoint, get the IP address of the web server container:

docker inspect webserver | grep IPAddress

On the Wazuh manager, check that it can see the container logs after a curl request is made:

tail -f /var/ossec/logs/archives/archives.json | grep webserver

From a machine that can reach the web server, do a curl request or browse to the web server container IP address:

curl <webserver_IP>

The curl request creates an event that is logged to the Wazuh manager archives:

{"timestamp":"2022-02-02T11:53:33.058+0000","agent":{"id":"025","name":"vagrant","ip":"10.0.2.15"},"manager":{"name":"wazuh-manager"},"id":"1643802813.1121072","full_log":"Feb  2 11:53:32 vagrant docker/webserver[24403]: 172.17.0.1 - - [02/Feb/2022:11:53:32 +0000] \"GET / HTTP/1.1\" 200 7228 \"-\" \"curl/7.58.0\" \"-\"","predecoder":{"program_name":"docker/webserver","timestamp":"Feb  2 11:53:32","hostname":"vagrant"},"decoder":{"name":"docker_log_decoder"},"data":{"srcip":"172.17.0.1","date":"02/Feb/2022","time":"2022:11:53","web_action":"\"GET / HTTP/1.1\" 200"},"location":"/var/log/syslog"}

The container logs are in the full_log field in the log above. This is the log of interest:

Feb  2 11:53:32 vagrant docker/webserver[24403]: 172.17.0.1 - - [02/Feb/2022:11:53:32 +0000] "GET / HTTP/1.1" 200 7228 "-" "curl/7.58.0" "-"'

Write decoders to extract information from the full_log field. To do it, edit /var/ossec/etc/decoders/local_decoder.xml on the Wazuh manager:

<decoder name="docker_log_decoder">
  <program_name>^docker</program_name>
</decoder>

<decoder name="docker_log_decoder_2">
  <parent>docker_log_decoder</parent>
  <regex offset="after_parent">(\d*.\d*.\d*.\d*)</regex>
  <order>srcip</order>
</decoder>

<decoder name="docker_log_decoder_2">
  <parent>docker_log_decoder</parent>
  <regex offset="after_parent">(\d*/\w*/\d*)</regex>
  <order>date</order>
</decoder>

<decoder name="docker_log_decoder_2">
  <parent>docker_log_decoder</parent>
  <regex offset="after_parent">(\d*:\d*:\d*)</regex>
  <order>time</order>
</decoder>

<decoder name="docker_log_decoder_2">
  <parent>docker_log_decoder</parent>
  <regex offset="after_parent">("\w*\s/\s\w*/\d.\d"\s\d*)</regex>
  <order>web_action</order>
</decoder>

We run the log test binary to check that the decoder is extracting the event fields:

/var/ossec/bin/wazuh-logtest

Paste the log after the binary starts:

**Phase 1: Completed pre-decoding.
        full event: 'Feb  2 11:53:32 vagrant docker/webserver[24403]: 172.17.0.1 - - [02/Feb/2022:11:53:32 +0000] "GET / HTTP/1.1" 200 7228 "-" "curl/7.58.0" "-"''
        timestamp: 'Feb  2 11:53:32'
        hostname: 'vagrant'
        program_name: 'docker/webserver'

**Phase 2: Completed decoding.
        name: 'docker_log_decoder'
        date: '02/Feb/2022'
        srcip: '172.17.0.1'
        time: '2022:11:53'
        web_action: '"GET / HTTP/1.1" 200'

To write a custom rule that triggers an alert when a GET request is made to the web server, edit /var/ossec/etc/rules/local_rules.xml on the Wazuh manager:

<group name="docker-log-rule,">
  <rule id="100009" level="5">
    <field name="web_action">"GET / HTTP/1.1" 200</field>
    <description>Docker Container: Successful Web GET request on Container: $(program_name) from $(srcip)</description> <mitre>
      <id>T1016.001</id>
    </mitre>
    <group>docker-log-rule,</group>
  </rule>
</group>

Restart the Wazuh manager for changes to apply:

systemctl restart wazuh-manager

To trigger alerts for the rule that was created, do a curl request or browse to the web server container IP address. In the security events tab of the Wazuh Dashboard, we can see an example of a successful GET request alert.

Monitoring Docker container logs with Wazuh

Conclusion

This article explains how Docker logging drivers combined with Rsyslog were used to group and tag Docker logs. The logs were then analyzed by Wazuh to generate useful alerts.

References