Monitoring application metrics such as error rate, memory usage, and CPU usage is useful for early anomaly detection and maintaining system health. These metrics offer insights into how software behaves under different conditions, such as during peak user activity, helping teams to identify issues early. While traditional monitoring often centers on the endpoint’s health, application-level metrics provide deeper insight into how the software is performing under varying conditions.

OpenTelemetry (OTel) is an open source observability framework that provides a standard way to generate, collect, and export telemetry data, including metrics, logs, and traces, directly from applications.

For telemetry data to be actionable, it needs to be collected, transformed, and routed to the appropriate platform. Data Prepper is an open source data ingestion and transformation pipeline that collects, filters, and enriches observability data before forwarding it to the target system. In this setup, Data Prepper transforms OpenTelemetry metrics into a format that Wazuh can parse and index alongside security events.

Ingesting application metrics into Wazuh enables teams to monitor both performance and security data from a single platform. This reduces the need to maintain multiple platforms and simplifies operational overhead.

This blog post demonstrates how to analyze application metrics into Wazuh using OpenTelemetry and Data Prepper. This approach enables you to analyze performance metrics with security events on the Wazuh platform.

Infrastructure

We use the following infrastructure to demonstrate OTel metrics ingestion into Wazuh using Data Prepper.

• A pre-built, ready-to-use Wazuh OVA 4.14.0. Follow this Wazuh OVA installation guide to download the virtual machine.
• An Ubuntu 24.04 endpoint which will serve as the application server, with:
  • Wazuh agent 4.14.0 installed and enrolled into the Wazuh server.
  • Docker and Docker Compose installed.

As shown above, the Wazuh agent forwards security events like intrusion detection, log analysis, vulnerability detection, and configuration assessment to the Wazuh server. An application instrumented with OTel standards exports metrics to the OpenTelemetry Collector. The OpenTelemetry Collector gathers performance metrics, which Data Prepper parses and forwards to the Wazuh indexer, enabling a unified platform for application performance and security.

Configuration

To demonstrate this ingestion, we first configure the Wazuh indexer to allow external communication with Data Prepper on port 9200. Then, we deploy an application instrumented with the OpenTelemetry standard, along with an OpenTelemetry Collector and Data Prepper, on the Ubuntu endpoint. Finally, we create the metrics-otel-v1* index on the Wazuh indexer using the Wazuh dashboard.

Wazuh server

Perform the following steps on the Wazuh server endpoint:

1. Update the network.host parameter in the /etc/wazuh-indexer/opensearch.yml configuration file:

network.host: "127.0.0.1,<WAZUH_INDEXER_IP>"

Replace <WAZUH_INDEXER_IP> with the IP address of the Wazuh Indexer.

This configuration allows the Wazuh indexer to listen on both the localhost interface and the specified server IP, enabling external connections and local access.

2. Restart the Wazuh indexer to apply the changes:

# systemctl restart wazuh-indexer

Wazuh dashboard

Perform the following steps on the Wazuh dashboard:

• Create an internal user
• Create a role and map the internal user

Create an internal user

We create an internal user to securely handle the connection between the Wazuh indexer and Data Prepper.

1. Navigate to Indexer management > Security > Internal users and click Create internal user.
2. Enter a username and a strong password in the username and password fields. This will be required later when configuring Data Prepper.
3. Click Create.

Create a role and map the internal user

We create a dedicated role and assign it specific privileges to control what Data Prepper can access within the Wazuh indexer.

1. Navigate to Indexer management > Security > Roles and click Create Role.
2. Enter otel-role in the Name field. Select cluster_all, indices:admin/template/get, and indices:admin/template/put in the Cluster permissions field.
3. Enter metrics-otel-v1* in the Index field. Select indices_all in the Index permissions field.
4. Click Add another index permission.
5. Enter .opendistro-ism-config in the Index field. Select indices_all in the Index permissions field.
6. Click Add another index permission.
7. Enter * in the Index field. Select manage_aliases in the Index permissions field.
8. Click Create.
9. Navigate to Mapped users and click Map users. 
10. Select the internal user you created and click Map.

Ubuntu endpoint

Perform the following steps on the Ubuntu endpoint:

• Generate SSL certificates
• Configure OpenTelemetry Collector
• Configure Data Prepper
• Configure and deploy services with Docker Compose

Generate SSL certificates

We generate SSL certificates to ensure secure communication between the OpenTelemetry Collector and Data Prepper.

1. Create a certs directory at /opt/data-prepper and switch into it:

# mkdir -p /opt/data-prepper/certs && cd /opt/data-prepper/certs

2. Create a private certificate authority (CA):

# openssl genrsa -out ca.key 4096
# openssl req -x509 -new -nodes -key ca.key -sha256 -days 365 \
  -subj "/CN=MyCompany Root CA" -out ca.crt

3. Generate a key for Data Prepper:

# openssl genrsa -out data-prepper.key 2048

4. Create a SAN (Subject Alternative Names) configuration:

# cat > san.cnf <<EOF
[req]
req_extensions = v3_req
distinguished_name = dn
[dn]
[v3_req]
subjectAltName = @alt_names
[alt_names]
DNS.1 = data-prepper
DNS.2 = localhost
EOF

5. Create a CSR (Certificate Signing Request):

# openssl req -new -key data-prepper.key \
  -subj "/CN=data-prepper" \
  -out data-prepper.csr

6. Sign the CSR:

# openssl x509 -req -in data-prepper.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out data-prepper.crt -days 365 -sha256 -extensions v3_req -extfile san.cnf

7. Grant read permissions to the data-prepper.crt file so that Data Prepper can use it:

# chmod 644 data-prepper.crt

Configure OpenTelemetry Collector

The OpenTelemetry Collector is responsible for collecting metrics from the instrumented application and shipping them to Data Prepper for further processing.

1. Create the OpenTelemetry Collector configuration file otel-collector-config.yaml in the /opt/data-prepper directory and insert the following:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:

exporters:
  otlp/data-prepper:
    endpoint: data-prepper:21892
    tls:
      ca_file: /etc/certs/ca.crt

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/data-prepper]

Where:

• endpoint: 0.0.0.0:4317 defines the endpoint on which the Open Telemetry collector will receive metrics.
• endpoint: data-prepper:21892 defines the endpoint to which the Open Telemetry collector will export metrics.

Configure Data Prepper

Configure Data Prepper to ingest, transform, and forward metrics to the Wazuh indexer.

1. Create the Data Prepper pipeline file pipeline.yaml in the /opt/data-prepper directory and insert the following:

entry-pipeline:
  source:
    otel_metrics_source:
      ssl: true
      sslKeyCertChainFile: /etc/certs/data-prepper.crt
      sslKeyFile: /etc/certs/data-prepper.key
      health_check_service: true
      port: 21892
  sink:
   - opensearch:
        hosts: ["https://<WAZUH_INDEXER_IP>:9200"]
        insecure: true
        username: "<WAZUH_INDEXER_USERNAME>"
        password: "<WAZUH_INDEXER_PASSWORD>"
        index_type: metric-analytics-plain

Replace:

• <WAZUH_INDEXER_IP> with the IP address of the Wazuh indexer.
• <WAZUH_INDEXER_USERNAME> with the internal user created here.
• <WAZUH_INDEXER_PASSWORD> with the password of the internal user created here.

Configure and deploy services with Docker Compose

We use Docker Compose to start the OpenTelemetry Collector, Data Prepper, and a sample application instrumented with the OpenTelemetry framework for testing.

1. Create the Docker compose file compose.yaml in the /opt/data-prepper directory and insert the following:

services:
  otel-collector:
    image: "otel/opentelemetry-collector-contrib:0.128.0"
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
      - ./certs/ca.crt:/etc/certs/ca.crt
    ports:
      - "4317:4317"
    depends_on:
      data-prepper:
        condition: service_healthy

  data-prepper:
    image: "opensearchproject/data-prepper:2.12.0"
    volumes:
      - ./pipeline.yaml:/usr/share/data-prepper/pipelines/pipelines.yaml
      - ./certs/data-prepper.crt:/etc/certs/data-prepper.crt
      - ./certs/data-prepper.key:/etc/certs/data-prepper.key
    ports:
      - "21892:21892"
    healthcheck:
      test: ["CMD-SHELL", "echo > /dev/tcp/localhost/21892"]
      interval: 30s
      timeout: 5s
      retries: 5

The compose.yaml file above includes the services described below:

• data-prepper: Acts as the central pipeline for ingesting, processing, and exporting telemetry data.
• otel-collector: Acts as an agent for collecting telemetry data.

2. Create a separate Docker compose file compose-sample-app.yaml in the /opt/data-prepper directory, with the following content to set up the demo application for monitoring:

services:
  sample-app:
    image: "jobri237/sample-flask-meter:1.0"
    ports:
      - "5000:5000"
    depends_on:
      otel-collector:
        condition: service_started

The compose-sample-app.yaml file above includes the services described below:

• sample-app: A metrics-generating application, used here for demonstration purposes. It exposes the following metrics:
  • http.server.request.count: Tracks the total number of incoming HTTP requests received by the application.
  • http.server.response.errors: Counts HTTP responses with status codes in the 4xx and 5xx ranges.
  • http.server.bytes_sent: Measures the volume of data sent back to clients in HTTP responses.
  • orders.created.total: A custom application metric that increments whenever the /order endpoint is called.
  • process.cpu.percent: Reports the CPU usage percentage for the running application.
  • system.memory.usage: Captures the total amount of system memory currently in use, reported in bytes.
  • system.load_average: Exports the system load average.
  • open.ports.count: Shows the number of ports currently in a listening state (TCP/UDP).
  • active_sessions: Maintains the count of in-memory sessions created through the API.

3. Start the services from the /opt/data-prepper directory:

# docker compose -f compose.yaml -f compose-sample-app.yaml up -d

Run the application

Perform the following steps on the Ubuntu endpoint to trigger metrics from the sample application.

1. Generate HTTP traffic to increase http.server.request.count and http.server.bytes_sent:

# curl http://localhost:5000/

2. Simulate order creation to increment the custom business metric orders.created.total:

# curl -X POST http://localhost:5000/order

3. Create a session to add a new entry in memory and update active_sessions:

# curl -X POST http://localhost:5000/session/create

The command output should be similar to the following:

{"session_id": "1733240923456"}

4. Delete the session using the returned session ID to decrease active_sessions:

# curl -X DELETE http://localhost:5000/session/delete/<SESSION_ID>

Replace <SESSION_ID> with the actual value from the creation step.

5. Trigger an error response on a non-existing path to increase http.server.response.errors:

# curl http://localhost:5000/does-not-exist

6. Simulate system load with repeated requests to increase process.cpu.percent, system.memory.usage, and system.load_average:

# for i in {1..50}; do curl -s http://localhost:5000/ > /dev/null; done

Viewing data on the Wazuh dashboard

In this section, we create the metrics-otel-v1* index in the Wazuh dashboard. We then build a saved search, create visualizations, and assemble a dashboard for a unified view of the application metrics.

Create an index pattern and a saved search

Perform the following steps on the Wazuh dashboard:

• Create the index pattern
• Create a saved search
• Create visualizations and dashboards

Create the index pattern

An index pattern tells the Wazuh dashboard which data fields are available and how to interpret them. Creating this index pattern ensures that Wazuh can discover and parse the OpenTelemetry metrics saved.

1. Navigate to Dashboards management > Dashboards Management > Index patterns on the Wazuh dashboard.
2. Click +Create index pattern and fill metrics-otel-v1* in the index pattern name field.
3. Click Next step, set time in the Time field drop-down list, and click Create index pattern.

Create a saved search

Applications can generate multiple metrics that operators or security teams might not initially be aware of. A saved search helps teams quickly review and understand which metrics the monitored application is sending before building dashboards.

1. Navigate to Explore > Discover and select metrics-otel-v1* as the index pattern name.

2. Click Available fields. Then click + Add field as column on the following fields: name, value, unit, kind, description, instrumentationScope.name.

Where: 

• name: identifies the metric name 
• value: is the metric measurement
• unit: is the unit of measurement
• kind: indicates the type of metric
• description: explains the metric
• instrumentationScope.name: shows the library or component produced by the metric

3. Click Save, fill otel-metrics as the Title, and click Save.

Create visualizations and dashboards

In this section, we create visualizations to understand better the metrics collected from our sample application.

You can create the following visualizations for each data type based on the relevant metrics for your monitored application. This approach applies to any monitored application by simply replacing the metric names with those generated by your application.

The following dashboard and visualizations are created based on the available fields in the saved search created above.

Click the upper-left menu icon ☰ and navigate to Explore > Dashboards > Create new Dashboard and add the following visualizations.

Gauge data type

Gauge metrics represent a value sampled at a specific point in time, such as CPU or memory usage. To visualize them in the Wazuh dashboard:

1. Click Create new and select the TSVB visualization on the New Visualization tab.
2. Navigate to Panel options:

• Set metrics-otel-v1* under Index name.
• Set time under Time field and >=1m under Interval.

3. Navigate to the Data > Metrics tab:

• Set Max under Aggregation and value under Field.
• Set Terms under Group by and instrumentationScope.name under By.

Process CPU usage

1. Navigate to Panel Options, and set name:process.cpu.percent under Panel filter.
2. Navigate to Data > Options, and set Percent under Data Formatter.
3. Click Save, set CPU usage under Title, and click Save and return.

Process memory usage

Repeat steps 1 to 3 under the Gauge data type section, then:

1. Navigate to Panel Options, and set name:system.memory.usage under Panel filter.
2. Navigate to Data > Options and set Bytes under Data Formatter.
3. Click Save, set Memory usage under Title, and click Save and return.

Number of active sessions

Repeat steps 1 to 3 under the Gauge data type section, then:

1. Navigate to Panel Options, and set name:active_sessions under Panel filter.
2. Navigate to the Metric tab, and set Active sessions under Label.
3. Click Save, set Active sessions under Title, and click Save and return.

Average load

Repeat steps 1 to 3 under the Gauge data type section, then:

1. Navigate to Panel Options, and set name:system.load_average under Panel filter.
2. Click Save, set Average load under Title, and click Save and return.

Ports in listening state

Repeat steps 1 to 3 under the Gauge data type section, then:

1. Navigate to Panel Options, and set name:open.ports.count under Panel filter.
2. Navigate to the Metric tab, and set TCP/UDP ports listening under Label.
3. Click Save, set ports listening under Title, and click Save and return.

Sum data type

1. Click Create new and select the TSVB visualization on the New Visualization tab.
2. Navigate to Panel options: 

• Set metrics-otel-v1* under Index name.
• Set time under Time field and >=1m under Interval.

3. Navigate to the Data > Metrics tab:

• Set Max under Aggregation and value under Field.
• Click +Add Metric, set Derivative under Aggregation, and Max of value under Metric
• Set Terms under Group by, and instrumentationScope.name under By.

HTTP server request count

1. Navigate to Panel Options, and set name:http.server.request.count under Panel filter.
2. Click Save, set Request count under Title, and click Save and return.

Server response size

Repeat steps 1 to 3, under the Sum data type section, then:

1. Navigate to Data > Options and set Bytes under Data Formatter.
2. Navigate to Panel Options, and set name:http.server.bytes_sent under Panel filter.
3. Click Save, set Server response size under Title, and click Save and return.

Orders created

Repeat steps 1 to 3 under the Sum data type section, then:

1. Navigate to Panel Options, and set name:orders.created.total under Panel filter.
2. Click Save, set Orders created under Title, and click Save and return.

Finally, click  Save and enter OTel metrics as the dashboard title, and then click Save. The GIF below shows the resulting dashboard:

Conclusion

Integrating OpenTelemetry with Wazuh through Data Prepper allows you to ingest and analyze application metrics alongside security data. This enables effective monitoring of both application performance and security from a single platform.

Wazuh is a free and open source security solution designed for threat detection, incident response, and compliance management. Wazuh continues to provide new integrations and capabilities, and has an active and supportive community. 

To learn more, explore our documentation and blog posts.

References

• Using OpenSearch to visualize metrics ingested with OpenTelemetry
• Metrics Data Model
• OpenSearch Data Prepper
• Otel Collector configuration