Monitoring Docker container logs with Wazuh

| by Francis Timilehin Jeremiah
Post icon

Introduction

By default, Docker container logs only show stdout and stderr standard streams, which are cleared when the container is destroyed. However, when non-interactive processes, like a database or web server are run, logs pertaining to these processes are generated. These logs can be collected using Docker logging drivers and saved to a file. If the design of a container is good enough, it can provide valuable logs to be examined. This blog post addresses collecting Docker container logs and using Wazuh to analyze them.

Example setup

  • Wazuh manager: It will take care of processing, through decoders and rules, the containers data collected by the agent.
  • Linux/Unix endpoint: This will host the Docker containers to be created and monitored. Wazuh agent, Docker, and Rsyslog are assumed to be installed on this system.

Configure Docker to use the Syslog logging driver (Linux/Unix endpoint)

By default, when a container is created by the docker engine, the JSON File logging driver is used. Logs captured this way can be found in /var/lib/docker/containers/[container-id]/[container-id]-json.log.

For this example, since the JSON File logging driver doesn’t allow the log file location to be changed, we will use the Syslog logging driver instead.

When creating a container, the --log-driver tag can be specified to set the log driver which will be used. A more global approach is to use a configuration that sets the logging driver for all containers created.

To set the global logging driver, create or edit /etc/docker/daemon.json. This way all containers created henceforth will use the Syslog driver:

{
  "log-driver": "syslog",
  "log-opts": {
    "syslog-address": "unixgram:///dev/log",
    "tag": "docker/{{.Name}}"
  }
}

The configuration above specifies:

  • The type of log driver used, which in this case is Syslog.
  • The log messages will be channeled to unixgram:///dev/log, then the Rsyslog daemon pulls data from that socket.
  • Log messages will be tagged with docker/{{.Name}}, where {{.Name}} replaces the container name.

Restart the Docker service to apply changes:

# systemctl restart docker

Configure Rsyslog (Linux/Unix endpoint)

We need to configure Rsyslog to create and tag new logs from containers in the /var/log/docker folder.

Let’s create a configuration file called wazuh-docker.conf in the /etc/rsyslog.d/ directory:

$FileCreateMode 0644
$template DockerDaemonLogFileName,"/var/log/docker/docker.log"
$template DockerContainerLogFileName,"/var/log/docker/%SYSLOGTAG:R,ERE,1,FIELD:docker/(.*)\[--end:secpath-replace%.log"
if $programname == 'dockerd' then {
  ?DockerDaemonLogFileName
  stop
}
if $programname == 'containerd' then {
  ?DockerDaemonLogFileName
  stop
}
if $programname == 'docker' then {
  if $syslogtag contains 'docker/' then {
  ?DockerContainerLogFileName
  stop
  }
}
$FileCreateMode 0600

The configuration above specifies:

  • The files created are in 0644 mode, meaning they will be readable by all the user groups, but writable by the user only.
  • Any log from dockerd, containerd, and docker runtime will be written to /var/log/docker/docker.log.
  • SYSLOGTAG:R,ERE,1,FIELD:docker/(.*)\[--end:secpath-replace% is equivalent to the {{.Name}} tag in the Docker daemon configuration. Therefore new logs will be located in /var/log/docker/[new container name].log.

Restart the Rsyslog service to apply changes:

# systemctl restart rsyslog

New logs should start appearing in the /var/log/docker directory when containers are created.

Testing the configuration with a web server container

Linux/Unix endpoint

To configure log collection locally on the agent, add the config below inside the <ossec_config> tag of the /var/ossec/etc/ossec.conf file. A wild card is used, this way the Wazuh agent will read every container log in that directory:

<localfile>
  <log_format>syslog</log_format>
  <location>/var/log/docker/*</location>
</localfile>

Restart the Wazuh agent for changes to apply:

# systemctl restart wazuh-agent

As mentioned earlier, non-interactive applications produce additional logs, depending on the design of the container. For example, a web server container will generate web logs. So we expect to see HTTP web requests like “GET” in the logs.

Let’s set up a simple web server that shows its hostname, IP address, and other information:

  • Create a container using the nginxdemos/hello container image:
# docker run -P -d --hostname webserver --name webserver nginxdemos/hello
  • Check that there is a new log file:
# ls /var/log/docker/webserver.log

Wazuh manager

Configure the manager to write the logs retrieved by the agent to /var/ossec/logs/archives/archives.json. These logs will be used to create rules and decoders. In the /var/ossec/etc/ossec.conf file, change <logall_json> to yes:

<logall_json>yes</logall_json>

Restart the Wazuh manager for changes to apply:

# systemctl restart wazuh-manager

Testing the configurations

On the Linux/Unix endpoint, get the IP address of the web server container:

# docker inspect webserver | grep IPAddress

On the Wazuh manager, check that it can see the container logs after a curl request is made:

# tail -f /var/ossec/logs/archives/archives.json | grep webserver

From a machine that can reach the web server, do a curl request or browse to the web server container IP address:

# curl <webserver_IP>
{"timestamp":"2022-02-02T11:53:33.058+0000","agent":{"id":"025","name":"vagrant","ip":"10.0.2.15"},"manager":{"name":"wazuh-manager"},"id":"1643802813.1121072","full_log":"Feb  2 11:53:32 vagrant docker/webserver[24403]: 172.17.0.1 - - [02/Feb/2022:11:53:32 +0000] \"GET / HTTP/1.1\" 200 7228 \"-\" \"curl/7.58.0\" \"-\"","predecoder":{"program_name":"docker/webserver","timestamp":"Feb  2 11:53:32","hostname":"vagrant"},"decoder":{"name":"docker_log_decoder"},"data":{"srcip":"172.17.0.1","date":"02/Feb/2022","time":"2022:11:53","web_action":"\"GET / HTTP/1.1\" 200"},"location":"/var/log/syslog"}

The full_log field is the container log extracted from the log above. This is the log of interest:

Feb  2 11:53:32 vagrant docker/webserver[24403]: 172.17.0.1 - - [02/Feb/2022:11:53:32 +0000] "GET / HTTP/1.1" 200 7228 "-" "curl/7.58.0" "-"'

Write decoders to extract information from the full_log field. To do it, edit /var/ossec/etc/decoders/local_decoder.xml on the Wazuh manager:

<decoder name="docker_log_decoder">
  <program_name>^docker</program_name>
</decoder>

<decoder name="docker_log_decoder_2">
  <parent>docker_log_decoder</parent>
  <regex offset="after_parent">(\d*.\d*.\d*.\d*)</regex>
  <order>srcip</order>
</decoder>

<decoder name="docker_log_decoder_2">
  <parent>docker_log_decoder</parent>
  <regex offset="after_parent">(\d*/\w*/\d*)</regex>
  <order>date</order>
</decoder>

<decoder name="docker_log_decoder_2">
  <parent>docker_log_decoder</parent>
  <regex offset="after_parent">(\d*:\d*:\d*)</regex>
  <order>time</order>
</decoder>

<decoder name="docker_log_decoder_2">
  <parent>docker_log_decoder</parent>
  <regex offset="after_parent">("\w*\s/\s\w*/\d.\d"\s\d*)</regex>
  <order>web_action</order>
</decoder>

Run the log test binary to check that the decoder is extracting the event fields:

# /var/ossec/bin/wazuh-logtest

Paste the log after the binary starts:

**Phase 1: Completed pre-decoding.
  full event: '{"timestamp":"2022-02-02T11:53:33.058+0000","agent":{"id":"025","name":"vagrant","ip":"10.0.2.15"},"manager":{"name":"wazuh-manager"},"id":"1643802813.1121072","full_log":"Feb  2 11:53:32 vagrant docker/webserver[24403]: 172.17.0.1 - - [02/Feb/2022:11:53:32 +0000] \"GET / HTTP/1.1\" 200 7228 \"-\" \"curl/7.58.0\" \"-\"","predecoder":{"program_name":"docker/webserver","timestamp":"Feb  2 11:53:32","hostname":"vagrant"},"decoder":{"name":"docker_log_decoder"},"data":{"srcip":"172.17.0.1","date":"02/Feb/2022","time":"2022:11:53","web_action":"\"GET / HTTP/1.1\" 200"},"location":"/var/log/syslog"}'

**Phase 2: Completed decoding.
  name: 'json'
  agent.id: '025'
  agent.ip: '10.0.2.15'
  agent.name: 'vagrant'
  data.date: '02/Feb/2022'
  data.srcip: '172.17.0.1'
  data.time: '2022:11:53'
  data.web_action: '"GET / HTTP/1.1" 200'
  decoder.name: 'docker_log_decoder'
  full_log: 'Feb  2 11:53:32 vagrant docker/webserver[24403]: 172.17.0.1 - - [02/Feb/2022:11:53:32 +0000] "GET / HTTP/1.1" 200 7228 "-" "curl/7.58.0" "-"'
  id: '1643802813.1121072'
  location: '/var/log/syslog'
  manager.name: 'wazuh-manager'
  predecoder.hostname: 'vagrant'
  predecoder.program_name: 'docker/webserver'
  predecoder.timestamp: 'Feb  2 11:53:32'
  timestamp: '2022-02-02T11:53:33.058+0000'

To write a custom rule that triggers an alert when a GET request is made to the web server, edit /var/ossec/etc/rules/local_rules.xml on the Wazuh manager:

<group name="docker-log-rule,">
  <rule id="100009" level="5">
    <field name="web_action">"GET / HTTP/1.1" 200</field>
    <description>Docker Container: Successful Web GET request on Container: $(program_name) from $(srcip)</description> <mitre>
      <id>T1016.001</id>
    </mitre>
    <group>docker-log-rule,</group>
  </rule>
</group>

Restart the Wazuh manager for changes to apply:

# systemctl restart wazuh-manager

To trigger alerts for the rule that was created, do a curl request or browse to the web server container IP address. In the security events tab of the Wazuh Dashboard, we can see an example of a successful GET request alert.

Monitoring Docker container logs with Wazuh

Conclusion

This article explains how Docker logging drivers combined with Rsyslog were used to group and tag Docker logs. The logs were then analyzed by Wazuh to generate useful alerts.

References