Managing multiple Wazuh clusters with Cross-Cluster Search

August 30th 2024 | by Samson Olugbenga Idowu | Wazuh 4.8.2

Cross-Cluster Search (CCS) in Wazuh allows alerts from remote Wazuh clusters to be queried and viewed at a centralized location. The centralized location known as the Cross-Cluster Search (CCS) environment is trusted by the remote Wazuh clusters, enabling it to perform search operations. This lets security alerts be seen via a single Wazuh dashboard at a remote Security Operations Center (SOC) while the log data stays at the source cluster.

Cross-Cluster Search helps organizations including Managed Security Service Providers (MSSP), to onboard their customers without compromising the confidentiality and integrity of the data.

This blog post demonstrates how to configure Cross-Cluster Search to manage multiple Wazuh clusters.

Scenario

Below is a problem statement and the proposed solutions to describe a use case for Cross-Cluster Search in Wazuh.

Problem statement

A cybersecurity operations company that provides managed Security Operations Center (SOC) service to customers wants to use the Wazuh SIEM and XDR. The company has two new customers, organizations A and B, who wish to subscribe to their managed SOC service.

The requirements are stated below:

Each customer’s log is kept separate, and access is tightly controlled.
Customer logs are retained within the customer environment.

Proposed solution

In our solution, we implement a Cross-Cluster Search (CCS) environment and two remote Wazuh clusters representing each organization. The clusters are categorized into CCS environment, cluster A, and cluster B, with the following cluster characteristics:

CCS environment

The CCS environment is only used to access and query security data from remote Wazuh clusters. Therefore, it compromises the Wazuh indexer and Wazuh dashboard with no Wazuh server or agents.
This is trusted by clusters A and B.

Cluster A

This comprises the Wazuh server and the Wazuh indexer. There is no Wazuh dashboard because visualization will be done in the CCS environment; however, the Wazuh dashboard can still be installed to view alerts locally.
Its Wazuh server (single node or Wazuh server cluster) collects and analyzes logs from Wazuh agents. It stores the data in the Wazuh indexer (single node or Wazuh indexer cluster) for organization A.
It trusts the CCS environment only.

Cluster B

This comprises the Wazuh server and the Wazuh indexer. There is no Wazuh dashboard because visualization will be done in the CCS environment; however, the Wazuh dashboard can still be installed to view alerts locally.
Its Wazuh server (single node or Wazuh server cluster) collects and analyzes logs from Wazuh agents. It stores the data in the Wazuh indexer (single node or Wazuh indexer cluster) for organization B.
It trusts the CCS environment only.

Infrastructure

We make use of the following infrastructure to demonstrate how to implement the Cross-Cluster Search (CCS) in Wazuh:

A Red Hat Enterprise Linux 9.4 server, hosting a Wazuh dashboard and Wazuh indexer for the CCS environment.
Two Red Hat Enterprise Linux 9.4 servers for cluster A. The first server will host the Wazuh server, while the second will host the Wazuh indexer.
Two Red Hat Enterprise Linux 9.4 servers for cluster B. The first server will host the Wazuh server, while the second will host the Wazuh indexer.

Note: The Wazuh components used are version 4.8.2.

The table below shows a further breakdown of the infrastructure.

Cluster	Component	Node name	IP address
CCS	Wazuh dashboard	ccs-wazuh-dashboard	192.168.186.60
CCS	Wazuh indexer	ccs-wazuh-indexer-1	192.168.186.60
Cluster A	Wazuh server	ca-wazuh-server-1	192.168.10.100
Cluster A	Wazuh indexer	ca-wazuh-indexer-1	192.168.10.101
Cluster B	Wazuh server	cb-wazuh-server-1	192.168.20.100
Cluster B	Wazuh indexer	cb-wazuh-indexer-1	192.168.20.101

Ensure network connectivity between the remote clusters and the CCS environment, irrespective of the network address.

Configuration

CCS environment

A Cross-Cluster Search (CCS) environment is authorized to perform search operations on remote Wazuh clusters. It is typically made up of the Wazuh indexer and the Wazuh dashboard. The Wazuh indexer node establishes trust with indexer nodes in remote clusters, enabling it to search for log data on these indexers. The data on the Wazuh indexer nodes in the remote clusters are displayed on the Wazuh dashboard in the CCS environment.

Generating certificates

Perform the following steps to generate the root certificate authority (CA), admin, and node certificates for the CCS Wazuh dashboard and indexer nodes.

1. Download the wazuh-certs-tool.sh script and the config.yml configuration file. The wazuh-certs-tool.sh script is used to generate certificates for the cluster. The config.yml file defines the IP addresses and node names of the Wazuh central components to be deployed.

# curl -sO https://packages.wazuh.com/4.8/wazuh-certs-tool.sh

# curl -sO https://packages.wazuh.com/4.8/config.yml

2. Edit the config.yml file and replace the name and IP variables with the corresponding node names and IP addresses:

nodes:
  # Wazuh indexer nodes
  indexer:
    - name: ccs-wazuh-indexer-1
      ip: "192.168.186.60"

  # Wazuh dashboard nodes
  dashboard:
    - name: ccs-wazuh-dashboard
      ip: "192.168.186.60"

Note: Comment out or remove the Wazuh server configuration since it is not in use.

3. Run the wazuh-certs-tool.sh script with option -A to create the root-ca, admin, and node certificates:

# bash ./wazuh-certs-tool.sh -A

25/08/2024 15:22:57 INFO: Generating the root certificate.
25/08/2024 15:22:57 INFO: Generating Admin certificates.
25/08/2024 15:22:58 INFO: Admin certificates created.
25/08/2024 15:22:58 INFO: Generating Wazuh indexer certificates.
25/08/2024 15:22:58 INFO: Wazuh indexer certificates created.
25/08/2024 15:22:58 INFO: Generating Wazuh dashboard certificates.
25/08/2024 15:22:59 INFO: Wazuh dashboard certificates created.

4. Copy the root CA certificate to the working directory of a node in cluster A and cluster B using the scp utility. The root CA is used later in the remote clusters to generate certificates.

# scp -r wazuh-certificates/root-ca.* <USER_NAME>@<IP_ADDRESS>:<DESTINATION_DIRECTORY>

Replace:

<USER_NAME> with the destination server’s username.
<IP_ADDRESS> with the destination server’s IP address.
<DESTINATION_DIRECTORY> with the destination server’s working directory.

5. Compress all the certificate files and remove the uncompressed version to allow for easier transfer to other component nodes within the CCS environment if need be:

# tar -cvf ./wazuh-certificates.tar -C ./wazuh-certificates/ .
# rm -rf ./wazuh-certificates

Wazuh indexer installation

Perform the following configuration steps on the Wazuh indexer node for the CCS environment.

1. Install the necessary dependencies:

# yum install -y coreutils

2. Import the Wazuh GPG key and add the Wazuh repository:

# rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH
# echo -e '[wazuh]\ngpgcheck=1\ngpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH\nenabled=1\nname=EL-$releasever - Wazuh\nbaseurl=https://packages.wazuh.com/4.x/yum/\nprotect=1' | tee /etc/yum.repos.d/wazuh.repo

3. Update the package manager:

# yum update -y

4. Install the Wazuh indexer package:

# yum -y install wazuh-indexer

5. Edit the /etc/wazuh-indexer/opensearch.yml configuration file:

network.host: "192.168.186.60"
node.name: "ccs-wazuh-indexer-1"
cluster.initial_master_nodes:
- "ccs-wazuh-indexer-1"
#- "node-2"
#- "node-3"
cluster.name: "ccs-cluster"
#discovery.seed_hosts:
#  - "node-1-ip"
#  - "node-2-ip"
#  - "node-3-ip"
node.max_local_storage_nodes: "3"
path.data: /var/lib/wazuh-indexer
path.logs: /var/log/wazuh-indexer

plugins.security.ssl.http.pemcert_filepath: /etc/wazuh-indexer/certs/indexer.pem
plugins.security.ssl.http.pemkey_filepath: /etc/wazuh-indexer/certs/indexer-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/wazuh-indexer/certs/indexer.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/wazuh-indexer/certs/indexer-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false
plugins.security.authcz.admin_dn:
- "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.enable_snapshot_restore_privilege: true

plugins.security.nodes_dn:
- "CN=ccs-wazuh-indexer-1,OU=Wazuh,O=Wazuh,L=California,C=US"
#- "CN=node-2,OU=Wazuh,O=Wazuh,L=California,C=US"
#- "CN=node-3,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-an$

### Option to allow Filebeat-oss 7.10.2 to work ###
compatibility.override_main_response_version: true

Replace the values of:

network.host with the IP address of the Wazuh indexer node.
node.name with the name of the Wazuh indexer node set in the config.yml file.
cluster.initial_master_nodes with the name of the Wazuh indexer node in the CCS environment.
cluster.name with the name of the cluster. For example, ccs-cluster.
plugins.security.nodes_dn with the distinguished name (DN) of the certificate of the CCS Wazuh indexer nodes. The common name (CN) is to be replaced with the node name of the CCS Wazuh indexer node provided in the config.yml file.

Note: Make sure that a copy of the wazuh-certificates.tar file created during the generating certificates step is placed in your current working directory.

6. Run the following commands, replacing <INDEXER_NODE_NAME> with the value of node.name configured in the /etc/wazuh-indexer/opensearch.yml file:

# NODE_NAME=<INDEXER_NODE_NAME>
# mkdir /etc/wazuh-indexer/certs
# tar -xf ./wazuh-certificates.tar -C /etc/wazuh-indexer/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./admin.pem ./admin-key.pem ./root-ca.pem
# mv -n /etc/wazuh-indexer/certs/$NODE_NAME.pem /etc/wazuh-indexer/certs/indexer.pem
# mv -n /etc/wazuh-indexer/certs/$NODE_NAME-key.pem /etc/wazuh-indexer/certs/indexer-key.pem
# chmod 500 /etc/wazuh-indexer/certs
# chmod 400 /etc/wazuh-indexer/certs/*
# chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs

7. Enable and start the Wazuh indexer service:

# systemctl daemon-reload
# systemctl enable wazuh-indexer
# systemctl start wazuh-indexer

8. Run the Wazuh indexer indexer-security-init.sh script on the Wazuh indexer node to load the new certificate information and initialize the Wazuh indexer:

# /usr/share/wazuh-indexer/bin/indexer-security-init.sh

**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.168.186.60:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.10.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: ccs-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index does not exists, attempt to create it ... done (0-all replicas)
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml
   SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml
   SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml
   SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success

9. Check that the Wazuh indexer is initialized and properly set up by running the following command:

# curl -k -u admin:admin https://<WAZUH_INDEXER_IP>:9200

{
  "name" : "ccs-wazuh-indexer-1",
  "cluster_name" : "ccs-cluster",
  "cluster_uuid" : "JkH8XuirS6m6PgdZFEQV2g",
  "version" : {
    "number" : "7.10.2",
    "build_type" : "rpm",
    "build_hash" : "eee49cb340edc6c4d489bcd9324dda571fc8dc03",
    "build_date" : "2023-09-20T23:54:29.889267151Z",
    "build_snapshot" : false,
    "lucene_version" : "9.7.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Wazuh dashboard installation

Perform the following configuration steps on the Wazuh dashboard node for the CCS environment.

1. Install the necessary dependencies:

# yum install libcap

2. Import the Wazuh GPG key and add the Wazuh repository if this was not done during the Wazuh indexer installation step:

# rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH
# echo -e '[wazuh]\ngpgcheck=1\ngpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH\nenabled=1\nname=EL-$releasever - Wazuh\nbaseurl=https://packages.wazuh.com/4.x/yum/\nprotect=1' | tee /etc/yum.repos.d/wazuh.repo

3. Update the package manager:

# yum update -y

4. Install the Wazuh dashboard package:

# yum -y install wazuh-dashboard

5. Edit the /etc/wazuh-dashboard/opensearch_dashboards.yml file to add the URL value of the Wazuh indexer in the opensearch.hosts section.

server.host: 0.0.0.0
   server.port: 443
   opensearch.hosts: "https://192.168.186.60:9200"
   opensearch.ssl.verificationMode: certificate

Note: Make sure that a copy of the wazuh-certificates.tar file created during the generating certificates step is placed in your working directory.

6. Replace <DASHBOARD_NODE_NAME> with your Wazuh dashboard node name, the same one set in config.yml when creating the certificates, and move the certificates to their corresponding directories:

# NODE_NAME=<DASHBOARD_NODE_NAME>
# mkdir /etc/wazuh-dashboard/certs
# tar -xf ./wazuh-certificates.tar -C /etc/wazuh-dashboard/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./root-ca.pem
# mv -n /etc/wazuh-dashboard/certs/$NODE_NAME.pem /etc/wazuh-dashboard/certs/dashboard.pem
# mv -n /etc/wazuh-dashboard/certs/$NODE_NAME-key.pem /etc/wazuh-dashboard/certs/dashboard-key.pem
# chmod 500 /etc/wazuh-dashboard/certs
# chmod 400 /etc/wazuh-dashboard/certs/*
# chown -R wazuh-dashboard:wazuh-dashboard /etc/wazuh-dashboard/certs

7. Enable and start the Wazuh dashboard service:

# systemctl daemon-reload
# systemctl enable wazuh-dashboard
# systemctl start wazuh-dashboard

8. Edit the /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml file and replace the hosts section with a similar configuration as below:

hosts:
  - Cluster A:
      url: https://192.168.10.100
      port: 55000
      username: wazuh-wui
      password: wazuh-wui
      run_as: true
  - Cluster B:
      url: https://192.168.20.100
      port: 55000
      username: wazuh-wui
      password: wazuh-wui
      run_as: true

Where:

url: This is the IP address value of the Wazuh server master node for the respective sites. Specify this for each site under the hosts section.
port: This is the Wazuh API communication port. By default, the value is set to 55000. Specify this for each site under the hosts section.
username: This is the username used to authenticate the connection to the Wazuh API. By default, the value is set to wazuh-wui. Specify this for each site under the hosts section.
password: This is the password used to authenticate the connection to the Wazuh API. By default, the value is set to wazuh-wui. Specify this for each site under the hosts section.
run_as: By default, this is set to false. Set the value to true for Wazuh server role mapping to take effect. Specify this for each site under the hosts section.

9. Restart the Wazuh dashboard service to apply the changes:

# systemctl restart wazuh-dashboard

Note: Accessing the Wazuh dashboard at this configuration stage will produce API errors since the Wazuh servers have not been installed yet. Complete all the steps before accessing the dashboard in the Set up Cross-Cluster Search section.

Cluster A

This remote Wazuh cluster collects, analyzes, and stores logs from Wazuh agents within the customer A environment. It consists of a Wazuh server that collects and analyzes log data from Wazuh agents and a Wazuh indexer that stores the Wazuh alerts. It can be configured as a single-node or multi-node Wazuh cluster.

The Wazuh indexer node in cluster A trusts only the Wazuh indexer node in the CCS environment, allowing it to query its data. Alerts and log data reside in cluster A, but the CCS environment can only query the data but not store it.

Generating certificates

Perform the following steps on a node with the root CA certificate within cluster A to generate the admin and node certificates.

Note: Make sure that a copy of the root-ca.key and root-ca.pem files created during the CCS generating certificates step are in your working directory.

1. Download the wazuh-certs-tool.sh script and the config.yml configuration file. The wazuh-certs-tool.sh script is used to generate certificates for the cluster. The config.yml file is used to define the IP addresses and node names of the Wazuh central components to be deployed:

# curl -sO https://packages.wazuh.com/4.8/wazuh-certs-tool.sh
# curl -sO https://packages.wazuh.com/4.8/config.yml

2. Edit the config.yml file and replace the name and IP values with the corresponding node name and IP address for the Wazuh server and indexer:

nodes:
  # Wazuh indexer nodes
  indexer:
    - name: ca-wazuh-indexer-1
      ip: "192.168.10.101"

  # Wazuh server nodes
  server:
    - name: ca-wazuh-server-1
      ip: "192.168.10.100"

Note: Comment out or remove the Wazuh dashboard configuration since it is not in use.

3. Run the wazuh-certs-tool.sh script with option -A and indicate the root-ca certificate and key created earlier to create the admin, and node certificates:

# bash ./wazuh-certs-tool.sh -A ./root-ca.pem ./root-ca.key

25/08/2024 16:20:31 INFO: Generating Admin certificates.
25/08/2024 16:20:31 INFO: Admin certificates created.
25/08/2024 16:20:31 INFO: Generating Wazuh indexer certificates.
25/08/2024 16:20:31 INFO: Wazuh indexer certificates created.
25/08/2024 16:20:31 INFO: Generating Filebeat certificates.
25/08/2024 16:20:32 INFO: Wazuh Filebeat certificates created.

4. Compress all the certificate files and remove the uncompressed version to allow for easier transfer to other component nodes within cluster A:

# tar -cvf ./wazuh-certificates.tar -C ./wazuh-certificates/ .
# rm -rf ./wazuh-certificates

5. Copy the wazuh-certificates.tar file to the Wazuh indexer and server nodes within cluster A using the scp utility:

# scp wazuh-certificates.tar <USER_NAME>@<IP_ADDRESS>:<DESTINATION_DIRECTORY>

Wazuh indexer installation

Perform the following configuration steps on the Wazuh indexer node for cluster A.

1. Install the necessary dependencies:

# yum install -y coreutils

2. Import the Wazuh GPG key and add the Wazuh repository:

# rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH
# echo -e '[wazuh]\ngpgcheck=1\ngpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH\nenabled=1\nname=EL-$releasever - Wazuh\nbaseurl=https://packages.wazuh.com/4.x/yum/\nprotect=1' | tee /etc/yum.repos.d/wazuh.repo

3. Update the package manager:

# yum update -y

4. Install the Wazuh indexer package:

# yum -y install wazuh-indexer

5. Edit the /etc/wazuh-indexer/opensearch.yml configuration file and replace the following values.

network.host with the IP address of the Wazuh indexer node.
node.name with the name of the Wazuh indexer node set in the config.yml file.
cluster.initial_master_nodes with the name of the Wazuh indexer node(s) in cluster A.
cluster.name with the name of the cluster. For example, ca-cluster.
plugins.security.nodes_dn with the distinguished name (DN) of the certificate for the cluster A Wazuh indexer node(s) and the CCS Wazuh indexer node. The common names (CN) will be replaced with the node names of the cluster A Wazuh indexer node and the CCS Wazuh indexer node.

Note: It is necessary to add the distinguished name (DN) of the CCS Wazuh indexer node in every remote Wazuh indexer node. This allows the remote clusters (clusters A and B) to trust the CCS environment.

network.host: "192.168.10.101"
node.name: "ca-wazuh-indexer-1"
cluster.initial_master_nodes:
- "ca-wazuh-indexer-1"
#- "node-2"
#- "node-3"
cluster.name: "ca-cluster"
#discovery.seed_hosts:
#  - "node-1-ip"
#  - "node-2-ip"
#  - "node-3-ip"
node.max_local_storage_nodes: "3"
path.data: /var/lib/wazuh-indexer
path.logs: /var/log/wazuh-indexer

plugins.security.ssl.http.pemcert_filepath: /etc/wazuh-indexer/certs/indexer.pem
plugins.security.ssl.http.pemkey_filepath: /etc/wazuh-indexer/certs/indexer-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/wazuh-indexer/certs/indexer.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/wazuh-indexer/certs/indexer-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false

plugins.security.authcz.admin_dn:
- "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.nodes_dn:
- "CN=ca-wazuh-indexer-1,OU=Wazuh,O=Wazuh,L=California,C=US"
- "CN=ccs-wazuh-indexer-1,OU=Wazuh,O=Wazuh,L=California,C=US"
#- "CN=node-3,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-an$

### Option to allow Filebeat-oss 7.10.2 to work ###
compatibility.override_main_response_version: true

Note: Make sure that a copy of the wazuh-certificates.tar file created during the generating certificates step is placed in your current working directory.

6. Run the following commands, replacing <INDEXER_NODE_NAME> with the value of node.name configured in the /etc/wazuh-indexer/opensearch.yml file:

# NODE_NAME=<INDEXER_NODE_NAME>
# mkdir /etc/wazuh-indexer/certs
# tar -xf ./wazuh-certificates.tar -C /etc/wazuh-indexer/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./admin.pem ./admin-key.pem ./root-ca.pem
# mv -n /etc/wazuh-indexer/certs/$NODE_NAME.pem /etc/wazuh-indexer/certs/indexer.pem
# mv -n /etc/wazuh-indexer/certs/$NODE_NAME-key.pem /etc/wazuh-indexer/certs/indexer-key.pem
# chmod 500 /etc/wazuh-indexer/certs
# chmod 400 /etc/wazuh-indexer/certs/*
# chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs

7. Enable and start the Wazuh indexer service:

# systemctl daemon-reload
# systemctl enable wazuh-indexer
# systemctl start wazuh-indexer

8. Run the Wazuh indexer indexer-security-init.sh script on the Wazuh indexer node to load the new certificate information and initialize the Wazuh indexer:

# /usr/share/wazuh-indexer/bin/indexer-security-init.sh

# /usr/share/wazuh-indexer/bin/indexer-security-init.sh

9. Check that the Wazuh indexer is initialized and correctly set up by running the following command:

# curl -k -u admin:admin https://<WAZUH_INDEXER_IP>:9200

{
  "name" : "ca-wazuh-indexer-1",
  "cluster_name" : "ca-cluster",
  "cluster_uuid" : "qsOmbP79SfqEM34sHH9I6g",
  "version" : {
    "number" : "7.10.2",
    "build_type" : "rpm",
    "build_hash" : "eee49cb340edc6c4d489bcd9324dda571fc8dc03",
    "build_date" : "2023-09-20T23:54:29.889267151Z",
    "build_snapshot" : false,
    "lucene_version" : "9.7.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Wazuh server installation

Perform the following configuration steps on the Wazuh server node for cluster A.

1. Import the Wazuh GPG key and add the Wazuh repository:

# rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH
# echo -e '[wazuh]\ngpgcheck=1\ngpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH\nenabled=1\nname=EL-$releasever - Wazuh\nbaseurl=https://packages.wazuh.com/4.x/yum/\nprotect=1' | tee /etc/yum.repos.d/wazuh.repo

2. Update the package manager:

# yum update -y

3. Install the Wazuh manager package:

# yum -y install wazuh-manager

4. Install the Filebeat package:

# yum -y install filebeat

5. Download the preconfigured Filebeat configuration file:

# curl -so /etc/filebeat/filebeat.yml https://packages.wazuh.com/4.8/tpl/wazuh/filebeat/filebeat.yml

6. Edit the /etc/filebeat/filebeat.yml configuration file and enter the IP address of the cluster A Wazuh indexer node(s) in the hosts section:

# Wazuh - Filebeat configuration file
output.elasticsearch:
  hosts: ["192.168.10.101:9200"]

Note: Only the IP addresses of cluster A Wazuh indexer nodes are to be entered in the hosts field above.

7. Create a Filebeat keystore to securely store authentication credentials and add the default username and password admin:admin to the secrets keystore:

# filebeat keystore create
# echo admin | filebeat keystore add username --stdin --force
# echo admin | filebeat keystore add password --stdin --force

8. Download the alerts template for the Wazuh indexer and grant appropriate read permissions:

# curl -so /etc/filebeat/wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/v4.8.1/extensions/elasticsearch/7.x/wazuh-template.json
# chmod go+r /etc/filebeat/wazuh-template.json

9. Install the Wazuh module for Filebeat:

# curl -s https://packages.wazuh.com/4.x/filebeat/wazuh-filebeat-0.4.tar.gz | tar -xvz -C /usr/share/filebeat/module

Note: Make sure that a copy of the wazuh-certificates.tar file created during the generating certificates step is placed in your working directory.

10. Replace <SERVER_NODE_NAME> with the cluster A Wazuh server node name set in config.yml when creating the certificates, and move the certificates to their corresponding directories:

# NODE_NAME=<SERVER_NODE_NAME>
# mkdir /etc/filebeat/certs
# tar -xf ./wazuh-certificates.tar -C /etc/filebeat/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./root-ca.pem
# mv -n /etc/filebeat/certs/$NODE_NAME.pem /etc/filebeat/certs/filebeat.pem
# mv -n /etc/filebeat/certs/$NODE_NAME-key.pem /etc/filebeat/certs/filebeat-key.pem
# chmod 500 /etc/filebeat/certs
# chmod 400 /etc/filebeat/certs/*
# chown -R root:root /etc/filebeat/certs

11. Save the Wazuh indexer username and password into the Wazuh manager keystore using the wazuh-keystore tool:

# /var/ossec/bin/wazuh-keystore -f indexer -k username -v <INDEXER_USERNAME>
# /var/ossec/bin/wazuh-keystore -f indexer -k password -v <INDEXER_PASSWORD>

12. Edit /var/ossec/etc/ossec.conf file to configure the indexer connection by adding the Wazuh indexer IP address to the <host> section of the <indexer> block:

<indexer>
  <enabled>yes</enabled>
  <hosts>
    <host>https://192.168.10.101:9200</host>
  </hosts>
  <ssl>
    <certificate_authorities>
      <ca>/etc/filebeat/certs/root-ca.pem</ca>
    </certificate_authorities>
    <certificate>/etc/filebeat/certs/filebeat.pem</certificate>
    <key>/etc/filebeat/certs/filebeat-key.pem</key>
  </ssl>
</indexer>

13. Enable and start the Wazuh manager service and Filebeat service:

# systemctl daemon-reload
# systemctl enable wazuh-manager
# systemctl start wazuh-manager
# systemctl enable filebeat
# systemctl start filebeat

14. Run the following command to verify that Filebeat is successfully installed:

# filebeat test output

elasticsearch: https://192.168.10.101:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 192.168.10.101
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.10.2

Cluster B

This is similar to cluster A and can be configured as a single-node or multi-node Wazuh cluster.

Perform the steps in cluster A to generate the certificates for cluster B Wazuh server and Wazuh indexer nodes and configure them. Use the unique node names and IP addresses dedicated to cluster B.

Adding Wazuh agents to the clusters

Follow the Wazuh agents installation guide to add Wazuh agents to either clusters A and B. The Wazuh agent connects to the Wazuh server within customer A or B environments.

Set up Cross-Cluster Search

Perform the following steps on the Wazuh dashboard to enable Cross-Cluster Search from the CCS environment to the remote clusters.

1. Log in to the Wazuh dashboard using the login credentials:

URL: https://<WAZUH_DASHBOARD_IP>
Username: admin
Password: admin

2. Select ☰ > Indexer management > DevTools and run the following API call to connect the CCS environment to the remote Wazuh clusters on port 9300:

Note: Add the Wazuh indexer node name for clusters A and B to the "cluster.remote" section and their corresponding IP addresses to the "seeds" section.

PUT _cluster/settings 
{
  "persistent": {
    "cluster.remote": {
      "ca-wazuh-indexer-1": {
        "seeds": ["192.168.10.101:9300"]
      },
      "cb-wazuh-indexer-1": {
        "seeds": ["192.168.20.101:9300"]
      }
    }
  }
}

{
  "acknowledged": true,
  "persistent": {
    "cluster": {
      "remote": {
        "cb-wazuh-indexer-1": {
          "seeds": [
            "192.168.20.101:9300"
          ]
        },
        "ca-wazuh-indexer-1": {
          "seeds": [
            "192.168.10.101:9300"
          ]
        }
      }
    }
  },
  "transient": {}
}

3. Test that the remote clusters are connected by running the following API call:

Note: Change the Wazuh indexer name highlighted to match the cluster being tested.

GET ca-wazuh-indexer-1:wazuh-alerts-*/_search

{
  "took": 833,
  "timed_out": false,
  "_shards": {
    "total": 6,
    "successful": 6,
    "skipped": 0,
    "failed": 0
  },
  "_clusters": {
    "total": 1,
    "successful": 1,
    "skipped": 0
  },
  "hits": {
    "total": {
      "value": 221,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "ca-wazuh-indexer-1:wazuh-alerts-4.x-2024.08.25",
        "_id": "BZ40i5EB-SZRRdc_oF6H",
        "_score": 1,
        "_source": {
          "predecoder": {
            "hostname": "ccs",
            "program_name": "systemd",
            "timestamp": "Aug 25 21:22:35"
          },
          "agent": {
            "name": "cluster-a",
            "id": "000"
          },
          "manager": {
            "name": "cluster-a"
          },
          "rule": {
            "firedtimes": 1,
            "mail": false,
            "level": 5,
            "description": "Systemd: System time has been changed.",
            "groups": [
              "local",
              "systemd"
            ],
            "id": "40705",
            "gpg13": [
              "4.3"
            ],
            "gdpr": [
              "IV_35.7.d"
            ]
          },
          "decoder": {
            "name": "systemd"
          },
          "full_log": "Aug 25 21:22:35 ccs systemd: Time has been changed",
          "input": {
            "type": "log"
          },
          "@timestamp": "2024-08-25T20:22:37.091Z",
          "location": "/var/log/messages",
          "id": "1724617357.562701",
          "timestamp": "2024-08-25T21:22:37.091+0100"
        }
      },
...

4. Select ☰ > Dashboard management > Dashboard Management > Index patterns and select Create index pattern to add the index patterns for the remote clusters.

5. Add the index pattern name using the format <NAME_OF_CLUSTER>:wazuh-alerts-* and select Next step. For example, ca-wazuh-indexer-1:wazuh-alerts-*.

6. Select timestamp as the primary time field.

7. Select Create index pattern to create the index pattern.

8. Navigate to the Wazuh home page and select the API and corresponding index pattern to view alerts. For example:

API: Cluster A
Index pattern: ca-wazuh-indexer-1:wazuh-alerts-*

Repeat steps 4 to 7 to create the index pattern for cluster B, cb-wazuh-indexer-1:wazuh-alerts-*.

Recommendations

Below are some suggestions to improve overall security and performance.

Ensure that there is a fast network connection between the CCS environment and the remote Wazuh clusters.
The CCS architecture can be scaled as desired to be single or multi-node.
Store the root-ca.key and root-ca.pem files in a secure location for future use, and remove them from all nodes after generating the certificates.
Remove the wazuh-certificates.tar file by running rm -f ./wazuh-certificates.tar if no other Wazuh component is to be installed on a node.
Change the default admin password by following our password management documentation.
Configure index retention policies to remove old indices from the Wazuh indexer to free up storage space. This is necessary to manage storage due to the replication and accumulation of indices on Wazuh indexer nodes.

Conclusion

Wazuh offers a unified security view across multiple organizations by enabling query of log data and security alerts from remote Wazuh clusters through Cross-Cluster Search (CCS). This approach allows you to monitor and respond to security events across multiple Wazuh deployments efficiently.

Additionally, managing multiple Wazuh clusters with Cross-Cluster Search encourages scalability and data residency close to the log source. This allows controlled access to an organization’s security events and logs while giving access to a third-party organization to view their security alerts.

If you have any questions or require assistance regarding this setup, refer to our community channels.

Reference

Managing multiple Wazuh clusters with Cross-Cluster Search

Problem statement

Proposed solution

CCS environment

Generating certificates

Wazuh indexer installation

Wazuh dashboard installation

Cluster A

Generating certificates

Wazuh indexer installation

Wazuh server installation

Cluster B

Adding Wazuh agents to the clusters

Set up Cross-Cluster Search

Related content

Scanning Docker infrastructure against CIS Benchmark with Wazuh

Ensuring NIS2 compliance with Wazuh

Daolpu infostealer detection and response with Wazuh

Achieving CJIS compliance with Wazuh