Kubernetes auditing offers insight into security-relevant events occurring in your system. It provides information about the sequence of activities that the different components have experienced over time.

Kube-apiserver takes care of analyzing every request and sends the event to a backend according to a previously defined policy.

There are three types of backends available:

  • Log backend writes events to a file.
  • Webhook backend sends events to an external API.
  • Dynamic backend configures a webhook backend through an AuditSink API object.

This post will focus on setting up a dynamic backend. Events will be sent to a Wazuh manager which will, in turn, trigger alerts in response.

Dynamic backends and AuditSink

Dynamic backends store audit events externally by configuring a webhook within an AuditSink API object.

To enable this type of backend you need to add the following Kubernetes API flags:

  • --audit-dynamic-configuration. This will be the only one needed once it is GA.
  • --feature-gates=DynamicAuditing=true
  • --runtime-config=auditregistration.k8s.io/v1alpha1=true

For testing purposes, you can run minikube (you can find an installation guide here):

$ sudo minikube start --vm-driver=none --apiserver-ips 127.0.0.1 --apiserver-name localhost --extra-config=apiserver.audit-dynamic-configuration=true --feature-gates=DynamicAuditing=true --extra-config=apiserver.runtime-config=auditregistration.k8s.io/v1alpha1

Now you can define the AuditSink object (auditsink.yml):

apiVersion: auditregistration.k8s.io/v1alpha1
kind: AuditSink
metadata:
    name: wazuh
spec:
    policy:
        level: Metadata
        stages:
            - ResponseComplete
    webhook:
        throttle:
            qps: 10
            burst: 15
        clientConfig:
            url: https://webhook_listener_url:webhook_listener_port/
            caBundle: your_base64_certificate

Note: You will need to specify your own clientConfig.url and clientConfig.caBundle. Please see below for more details.

There are two major stanzas in the AuditSink spec.

Policy

The audit policy defines rules addressing which events should be considered to be sent to the webhook listener.

The first rule sets the audit level:

  • None. Don’t log events that match this rule.
  • Metadata. Log request metadata (requesting user, timestamp, resource, verb, etc.) but not request or response body.
  • Request. Log event metadata and request body but not response body. This does not apply for non-resource requests.
  • RequestResponse. Log event metadata, request and response bodies. This does not apply for non-resource requests.

The second one refers to the stage of the execution:

  • RequestReceived. The stage for events generated as soon as the audit handler receives the request, and before it is delegated down the handler chain.
  • ResponseStarted. This stage is only generated for long-running requests once the response headers are sent, but before the response body is sent.
  • ResponseComplete. The response body has been completed and no more bytes will be sent.
  • Panic. Events generated when a panic occurred.

The example above is using events in the Metadata level whenever its stage is ResponseComplete.

You can read more about its definition here.

Webhook

The following information is needed for the dynamic backend to contact the webhook listener. The two main fields are:

  • clientConfig.caBundle. It is a base64 representation of your webhook listener certificate.
  • clientConfig.url. This is the url where you will run the webhook listener (your manager instance).

Generating the certificate

You need to specify a certificate for Kubernetes to authenticate the webhook listener.

First, create a configuration file and fill it in with your information:

$ cat  > csr.conf  <<\EOF
[ req ]
prompt = no
default_bits = 2048
default_md = sha256
distinguished_name = req_distinguished_name
x509_extensions = v3_req

[req_distinguished_name]
C = US
ST = California
L = San Jose
O = Wazuh
OU = Research and development
emailAddress = javier@wazuh.com
CN = webhook_listener_server_ip

[ v3_req ]
authorityKeyIdentifier=keyid,issuer
basicConstraints = CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
IP.1 = webhook_listener_server_ip
IP.2 = alternative_webhook_listener_server_ip
EOF

Create rootCA public and private keys:

$ openssl req -x509 -new -nodes -newkey rsa:2048 -keyout rootCA.key -out rootCA.pem -batch -subj "/C=US/ST=California/L=San Jose/O=Wazuh"

Create csr and server private key:

$ openssl req -new -nodes -newkey rsa:2048 -keyout server.key -out server.csr -config csr.conf

Generate the server certificate:

$ openssl x509 -req -in server.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out server.crt -extfile csr.conf -extensions v3_req

Transform to base64:

$ openssl base64 -in server.crt -out base64-output -A

This base64 output is needed in the clientConfig.caBundle field within the AuditSink yml file.

Applying the AutidSink to your Kubernetes environment is very simple (kubectl installation guide here):

$ sudo kubectl apply -f auditsink.yml

The webhook listener

The webhook listener is designed to run on the Wazuh manager side and it is tasked with processing and sending the Kubernetes audit logs to the manager via socket:

#!/usr/bin/env python

# Copyright (C) 2015-2019, Wazuh Inc.
# Created by Wazuh, Inc. .
# This program is a free software; you can redistribute it and/or modify it under the terms of GPLv2

import json
import sys
import logging
import BaseHTTPServer, SimpleHTTPServer
import ssl
from socket import socket, AF_UNIX, SOCK_DGRAM

########################## Global variables ##########################

# Analysisd socket address
socket_addr = '/var/ossec/queue/ossec/queue'

# Webhook listener address and port
listener_address = 'webhook_listener_url'
listener_port = webhook_listener_port

# Webhook listener certificates
ssl_cert = '/path/to/webhook_listener.crt'
ssl_key = '/path/to/webhook_listener.key

########################## Common functions ##########################

# Send event to analysisd socket
def send_event(msg):
    string = '1:k8s:{0}'.format(json.dumps(msg))
    sock = socket(AF_UNIX, SOCK_DGRAM)
    sock.connect(socket_addr)
    sock.send(string.encode())
    sock.close()

# Define POST function for listener
class ServerHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
    def do_POST(self):
        # Get body
        content_len = int(self.headers.getheader('content-length', 0))
        post_body = self.rfile.read(content_len)

        # Build event and send it to analysisd
        k8s_event = {}
        data = json.loads(post_body)
        items = data['items']
        for item in items:
            k8s_event['k8s'] = item
            send_event(k8s_event)

        # Send response
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()

########################## Main workflow ##########################

if __name__ == '__main__':
    # Start logging config
    logging.basicConfig(level=logging.INFO, format='%(asctime)s: [%(levelname)s] %(message)s', datefmt="%Y-%m-%d %H:%M:%S",)

    try:
        # Create listener
        httpd = BaseHTTPServer.HTTPServer((listener_address, listener_port), ServerHandler)

        # Add certificates
        httpd.socket = ssl.wrap_socket (httpd.socket, server_side=True, keyfile=ssl_key, certfile=ssl_cert)

        # Run listener
        httpd.serve_forever()
    except Exception as e:
        logging.error("Error running webhook listener: {}".format(e))
        sys.exit()

The script uses native python libraries to run a simple http server that handles the POST request sent by the Kubernetes dynamic backend:

# Define POST function for listener
class ServerHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
    def do_POST(self):
        # Get body
        content_len = int(self.headers.getheader('content-length', 0))
        post_body = self.rfile.read(content_len)

        # Build event and send it to analysisd
        k8s_event = {}
        data = json.loads(post_body)
        items = data['items']
        for item in items:
            k8s_event['k8s'] = item
            send_event(k8s_event)

        # Send response
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()

A function sends every event to an analysisd socket:

# Send event to analysisd socket
def send_event(msg):
    string = '1:k8s:{0}'.format(json.dumps(msg))
    sock = socket(AF_UNIX, SOCK_DGRAM)
    sock.connect(socket_addr)
    sock.send(string.encode())
    sock.close()

Note: Make sure you change listener_address, listener_port, ssl_cert and ssl_key variables accordingly.

Then place the script somewhere in your Wazuh manager server and execute it:

$ python /path/to/webhook_listener.py

Note: The webhook listener is an HTTP server. You may need to run it in the background by using nohup or something similar.

Wazuh manager configuration

Kubernetes audit logs conform to the JSON schema and Wazuh will automatically decode them.
At this point you only need to define rules; place this in /var/ossec/etc/rules/local_rules.xml:

<group name="k8s_audit,">
	<rule id="100300" level="5">
		<location>k8s</location>
		<field name="k8s.stage">ResponseComplete</field>
		<options>no_full_log</options>
		<description>Kubernetes audit log $(k8s.objectRef.resource) resource $(k8s.verb) operation.</description>
	</rule>
</group>

Don’t forget to restart the Wazuh manager afterwards.

Examples

These are just some examples of the type of information you can obtain by auditing Kubernetes with Wazuh, but there’s much more to it as any resource type within Kubernetes will generate events over time.

Just for reference, you can find a list of resource types here

Creating a deployment

You can run this in minikube. It consists of an existing image named echoserver:

$ sudo kubectl create deployment hello-minikube --image=k8s.gcr.io/echoserver:1.10

This is what the alert will look like in Wazuh:

{
	"timestamp": "2019-10-25T09:43:43.853+0000",
	"rule": {
		"level": 5,
		"description": "Kubernetes audit log deployments resource create operation.",
		"id": "100300",
		"firedtimes": 4308,
		"mail": false,
		"groups": ["k8s_audit"]
	},
	"agent": {
		"id": "000",
		"name": "vagrant"
	},
	"manager": {
		"name": "vagrant"
	},
	"id": "1571996623.10119583",
	"decoder": {
		"name": "json"
	},
	"data": {
		"k8s": {
			"objectRef": {
				"namespace": "default",
				"resource": "deployments",
				"name": "hello-minikube",
				"apiVersion": "v1",
				"apiGroup": "apps"
			},
			"sourceIPs": "::1,",
			"level": "Metadata",
			"requestURI": "/apis/apps/v1/namespaces/default/deployments",
			"stageTimestamp": "2019-10-25T09:43:16.723515Z",
			"annotations": {
				"authorization": {
					"k8s": {
						"io/decision": "allow"
					}
				}
			},
			"auditID": "0a8ce3a7-0d80-47d4-a970-9dec642a7337",
			"requestReceivedTimestamp": "2019-10-25T09:43:16.578815Z",
			"verb": "create",
			"user": {
				"username": "minikube-user",
				"groups": "system:masters,system:authenticated,"
			},
			"userAgent": "kubectl/v1.16.2 (linux/amd64) kubernetes/c97fe50",
			"responseStatus": {
				"code": "201"
			},
			"stage": "ResponseComplete"
		}
	},
	"location": "k8s"
}

Listing pods

Execute this in your Kubernetes environment:

$ sudo kubectl get pods

This is a sample of the resulting alert:

{
	"timestamp": "2019-10-25T09:49:23.168+0000",
	"rule": {
		"level": 5,
		"description": "Kubernetes audit log pods resource list operation.",
		"id": "100300",
		"firedtimes": 7864,
		"mail": false,
		"groups": ["k8s_audit"]
	},
	"agent": {
		"id": "000",
		"name": "vagrant"
	},
	"manager": {
		"name": "vagrant"
	},
	"id": "1571996963.16692991",
	"decoder": {
		"name": "json"
	},
	"data": {
		"k8s": {
			"objectRef": {
				"namespace": "default",
				"resource": "pods",
				"apiVersion": "v1"
			},
			"sourceIPs": "::1,",
			"level": "Metadata",
			"requestURI": "/api/v1/namespaces/default/pods?limit=500",
			"stageTimestamp": "2019-10-25T09:49:13.572906Z",
			"annotations": {
				"authorization": {
					"k8s": {
						"io/decision": "allow"
					}
				}
			},
			"auditID": "c8d5b5f1-a16c-4903-acc4-6ce640fcc8a8",
			"requestReceivedTimestamp": "2019-10-25T09:49:13.537383Z",
			"verb": "list",
			"user": {
				"username": "minikube-user",
				"groups": "system:masters,system:authenticated,"
			},
			"userAgent": "kubectl/v1.16.2 (linux/amd64) kubernetes/c97fe50",
			"responseStatus": {
				"code": "200"
			},
			"stage": "ResponseComplete"
		}
	},
	"location": "k8s"
}

References