Auditing Kubernetes with Wazuh

Kubernetes auditing offers insight into security-relevant events occurring in your system. It provides information about the sequence of activities that the different components have experienced over time.
Kube-apiserver
takes care of analyzing every request and sends the event to a backend according to a previously defined policy.
There are three types of backends available:
Log backend
writes events to a file.Webhook backend
sends events to an external API.Dynamic backend
configures a webhook backend through an AuditSink API object.This post will focus on setting up a dynamic backend. Events will be sent to a Wazuh manager which will, in turn, trigger alerts in response.
Dynamic backends store audit events externally by configuring a webhook within an AuditSink API object.
To enable this type of backend you need to add the following Kubernetes API flags:
--audit-dynamic-configuration
. This will be the only one needed once it is GA.--feature-gates=DynamicAuditing=true
--runtime-config=auditregistration.k8s.io/v1alpha1=true
For testing purposes, you can run minikube (you can find an installation guide here):
sudo minikube start --vm-driver=none --apiserver-ips 127.0.0.1 --apiserver-name localhost --extra-config=apiserver.audit-dynamic-configuration=true --feature-gates=DynamicAuditing=true --extra-config=apiserver.runtime-config=auditregistration.k8s.io/v1alpha1
Now you can define the AuditSink object (auditsink.yml
):
apiVersion: auditregistration.k8s.io/v1alpha1 kind: AuditSink metadata: name: wazuh spec: policy: level: Metadata stages: - ResponseComplete webhook: throttle: qps: 10 burst: 15 clientConfig: url: https://webhook_listener_url:webhook_listener_port/ caBundle: your_base64_certificate
Note: You will need to specify your own clientConfig.url
and clientConfig.caBundle
. Please see below for more details.
There are two major stanzas in the AuditSink spec.
The audit policy defines rules addressing which events should be considered to be sent to the webhook listener.
The first rule sets the audit level:
None
. Don’t log events that match this rule.Metadata
. Log request metadata (requesting user, timestamp, resource, verb, etc.) but not request or response body.Request
. Log event metadata and request body but not response body. This does not apply for non-resource requests.RequestResponse
. Log event metadata, request and response bodies. This does not apply for non-resource requests.The second one refers to the stage of the execution:
RequestReceived
. The stage for events generated as soon as the audit handler receives the request, and before it is delegated down the handler chain.ResponseStarted
. This stage is only generated for long-running requests once the response headers are sent, but before the response body is sent.ResponseComplete
. The response body has been completed and no more bytes will be sent.Panic
. Events generated when a panic occurred.The example above is using events in the Metadata
level whenever its stage is ResponseComplete
.
The following information is needed for the dynamic backend to contact the webhook listener. The two main fields are:
clientConfig.caBundle
. It is a base64 representation of your webhook listener certificate.clientConfig.url
. This is the url where you will run the webhook listener (your manager instance).You need to specify a certificate for Kubernetes to authenticate the webhook listener.
First, create a configuration file and fill it in with your information:
cat > csr.conf <<\EOF [ req ] prompt = no default_bits = 2048 default_md = sha256 distinguished_name = req_distinguished_name x509_extensions = v3_req [req_distinguished_name] C = US ST = California L = San Jose O = Wazuh OU = Research and development emailAddress = javier@wazuh.com CN = webhook_listener_server_ip [ v3_req ] authorityKeyIdentifier=keyid,issuer basicConstraints = CA:FALSE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment subjectAltName = @alt_names [alt_names] IP.1 = webhook_listener_server_ip IP.2 = alternative_webhook_listener_server_ip EOF
Create rootCA public and private keys:
openssl req -x509 -new -nodes -newkey rsa:2048 -keyout rootCA.key -out rootCA.pem -batch -subj "/C=US/ST=California/L=San Jose/O=Wazuh"
Create csr and server private key:
openssl req -new -nodes -newkey rsa:2048 -keyout server.key -out server.csr -config csr.conf
Generate the server certificate:
openssl x509 -req -in server.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out server.crt -extfile csr.conf -extensions v3_req
Transform to base64:
openssl base64 -in server.crt -out base64-output -A
This base64 output is needed in the clientConfig.caBundle
field within the AuditSink yml file.
Applying the AutidSink to your Kubernetes environment is very simple (kubectl installation guide here):
$ sudo kubectl apply -f auditsink.yml
The webhook listener is designed to run on the Wazuh manager side and it is tasked with processing and sending the Kubernetes audit logs to the manager via socket:
#!/usr/bin/env python # Copyright (C) 2015-2019, Wazuh Inc. # Created by Wazuh, Inc. . # This program is a free software; you can redistribute it and/or modify it under the terms of GPLv2 import json import sys import logging import BaseHTTPServer, SimpleHTTPServer import ssl from socket import socket, AF_UNIX, SOCK_DGRAM ########################## Global variables ########################## # Analysisd socket address socket_addr = '/var/ossec/queue/ossec/queue' # Webhook listener address and port listener_address = 'webhook_listener_url' listener_port = webhook_listener_port # Webhook listener certificates ssl_cert = '/path/to/webhook_listener.crt' ssl_key = '/path/to/webhook_listener.key' ########################## Common functions ########################## # Send event to analysisd socket def send_event(msg): string = '1:k8s:{0}'.format(json.dumps(msg)) sock = socket(AF_UNIX, SOCK_DGRAM) sock.connect(socket_addr) sock.send(string.encode()) sock.close() # Define POST function for listener class ServerHandler(SimpleHTTPServer.SimpleHTTPRequestHandler): def do_POST(self): # Get body content_len = int(self.headers.getheader('content-length', 0)) post_body = self.rfile.read(content_len) # Build event and send it to analysisd k8s_event = {} data = json.loads(post_body) items = data['items'] for item in items: k8s_event['k8s'] = item send_event(k8s_event) # Send response self.send_response(200) self.send_header("Content-type", "text/html") self.end_headers() ########################## Main workflow ########################## if __name__ == '__main__': # Start logging config logging.basicConfig(level=logging.INFO, format='%(asctime)s: [%(levelname)s] %(message)s', datefmt="%Y-%m-%d %H:%M:%S",) try: # Create listener httpd = BaseHTTPServer.HTTPServer((listener_address, listener_port), ServerHandler) # Add certificates httpd.socket = ssl.wrap_socket (httpd.socket, server_side=True, keyfile=ssl_key, certfile=ssl_cert) # Run listener httpd.serve_forever() except Exception as e: logging.error("Error running webhook listener: {}".format(e)) sys.exit()
The script uses native python libraries to run a simple http server that handles the POST request sent by the Kubernetes dynamic backend:
# Define POST function for listener class ServerHandler(SimpleHTTPServer.SimpleHTTPRequestHandler): def do_POST(self): # Get body content_len = int(self.headers.getheader('content-length', 0)) post_body = self.rfile.read(content_len) # Build event and send it to analysisd k8s_event = {} data = json.loads(post_body) items = data['items'] for item in items: k8s_event['k8s'] = item send_event(k8s_event) # Send response self.send_response(200) self.send_header("Content-type", "text/html") self.end_headers()
A function sends every event to an analysisd socket:
# Send event to analysisd socket def send_event(msg): string = '1:k8s:{0}'.format(json.dumps(msg)) sock = socket(AF_UNIX, SOCK_DGRAM) sock.connect(socket_addr) sock.send(string.encode()) sock.close()
Note: Make sure you change listener_address
, listener_port
, ssl_cert
and ssl_key
variables accordingly.
Then place the script somewhere in your Wazuh manager server and execute it:
$ python /path/to/webhook_listener.py
Note: The webhook listener is an HTTP server. You may need to run it in the background by using nohup or something similar.
Kubernetes audit logs conform to the JSON schema and Wazuh will automatically decode them.
At this point you only need to define rules; place this in /var/ossec/etc/rules/local_rules.xml
:
<group name="k8s_audit,"> <rule id="100300" level="5"> <location>k8s</location> <field name="k8s.stage">ResponseComplete</field> <options>no_full_log</options> <description>Kubernetes audit log $(k8s.objectRef.resource) resource $(k8s.verb) operation.</description> </rule> </group>
Don’t forget to restart the Wazuh manager afterwards.
These are just some examples of the type of information you can obtain by auditing Kubernetes with Wazuh, but there’s much more to it as any resource type within Kubernetes will generate events over time.
Just for reference, you can find a list of resource types here
You can run this in minikube. It consists of an existing image named echoserver
:
$ sudo kubectl create deployment hello-minikube --image=k8s.gcr.io/echoserver:1.10
This is what the alert will look like in Wazuh:
{ "timestamp": "2019-10-25T09:43:43.853+0000", "rule": { "level": 5, "description": "Kubernetes audit log deployments resource create operation.", "id": "100300", "firedtimes": 4308, "mail": false, "groups": ["k8s_audit"] }, "agent": { "id": "000", "name": "vagrant" }, "manager": { "name": "vagrant" }, "id": "1571996623.10119583", "decoder": { "name": "json" }, "data": { "k8s": { "objectRef": { "namespace": "default", "resource": "deployments", "name": "hello-minikube", "apiVersion": "v1", "apiGroup": "apps" }, "sourceIPs": "::1,", "level": "Metadata", "requestURI": "/apis/apps/v1/namespaces/default/deployments", "stageTimestamp": "2019-10-25T09:43:16.723515Z", "annotations": { "authorization": { "k8s": { "io/decision": "allow" } } }, "auditID": "0a8ce3a7-0d80-47d4-a970-9dec642a7337", "requestReceivedTimestamp": "2019-10-25T09:43:16.578815Z", "verb": "create", "user": { "username": "minikube-user", "groups": "system:masters,system:authenticated," }, "userAgent": "kubectl/v1.16.2 (linux/amd64) kubernetes/c97fe50", "responseStatus": { "code": "201" }, "stage": "ResponseComplete" } }, "location": "k8s" }
Execute this in your Kubernetes environment:
$ sudo kubectl get pods
This is a sample of the resulting alert:
{ "timestamp": "2019-10-25T09:49:23.168+0000", "rule": { "level": 5, "description": "Kubernetes audit log pods resource list operation.", "id": "100300", "firedtimes": 7864, "mail": false, "groups": ["k8s_audit"] }, "agent": { "id": "000", "name": "vagrant" }, "manager": { "name": "vagrant" }, "id": "1571996963.16692991", "decoder": { "name": "json" }, "data": { "k8s": { "objectRef": { "namespace": "default", "resource": "pods", "apiVersion": "v1" }, "sourceIPs": "::1,", "level": "Metadata", "requestURI": "/api/v1/namespaces/default/pods?limit=500", "stageTimestamp": "2019-10-25T09:49:13.572906Z", "annotations": { "authorization": { "k8s": { "io/decision": "allow" } } }, "auditID": "c8d5b5f1-a16c-4903-acc4-6ce640fcc8a8", "requestReceivedTimestamp": "2019-10-25T09:49:13.537383Z", "verb": "list", "user": { "username": "minikube-user", "groups": "system:masters,system:authenticated," }, "userAgent": "kubectl/v1.16.2 (linux/amd64) kubernetes/c97fe50", "responseStatus": { "code": "200" }, "stage": "ResponseComplete" } }, "location": "k8s" }
If you have any questions about this, don’t hesitate to check out our documentation to learn more about Wazuh or join our community where our team and contributors will help you.