Wazuh index management
Wazuh helps you comply with the security standards in which logs are required to be maintained for several months so that they can be provided on the spot in case of an audit.
Being able to quickly access all this information requires storing it on hard disks. Since storage space has a cost and a limit, you may have to delete old data to ensure you can maintain the retention period that you need.
Alerts generated by Wazuh are sent to an Elasticsearch daily index named wazuh-alerts-3.x-YYYY.MM.DD by using the default configuration.
In this blog post you will learn how to configure Elastic ILM and OpenDistro ISM to automatically manage the data in those indices over time.
Elastic ILM
You can create policies that govern what is the lifecycle of the indices based on different phases.
Four phases can be defined in a Lifecycle Policy:
Hot phase. For recent data that is actively accessed.Warm phase. Data that you may wish to access, but less often.Cold phase. Similar to the warm phase but you may also freeze indices to reduce overhead.Delete phase. Data that reaches this phase is deleted.
Create a policy
To configure an index lifecycle policy you may go into the Management section of your Kibana web interface, select Index Lifecycle Policies and then click on the Create policy button:
Once there you can specify the various options of the policy:
- Provide a name to the policy.
- You may disable the rollover option unless you wish to use it. For more information see using rollover policies.
- You can reduce the overhead of an index by enabling a cold phase.
- Specify a time after the cold phase will be applied.
- Activate the delete phase.
- Specify the age of the index before it is deleted.
- Finally, save your new policy.
Note: If you have elasticsearch nodes that have hardware of lower performance and cost, you may specify that during the cold phase the data will be stored on these nodes. For more information see shard allocation.
Add a policy to an index template
The next step is to apply this new policy to the index template of Wazuh alerts. In order to do so unfurl the Actions menu and select Add policy to index template:
Then select wazuh from the index template drop-down menu, and click on Add policy:
This will apply the policy to all wazuh alerts indices created in the future.
To apply this to already existing indices you can use an API call or the Index Management tool.
Using the Index Management tool
You may search and select the indices to which you wish to apply the policy, then select Add lifecycle policy from the Manage index menu:
Once there select the newly created policy and click on Add policy:
Using the Elasticsearch API
Alternatively, you may use an API call to apply this setting to all wazuh alert indices. Paste the following:
PUT wazuh-alerts-3.x-*/_settings
{
"index.lifecycle.name":"wazuh-alert-retention-policy"
}
into the Kibana Dev Tools console, and click on the triangle to send the request:
The system will reply with {"acknowledged" : true}.
With this, the Elastic ILM configuration is completed.
OpenDistro ISM
Open Distro is a fully open-source project maintained by Amazon Web Services which aims to provide an alternative to the proprietary features of Elasticsearch. Although very similar, there are some key differences, one of which is how to automatically manage the lifecycle of indices.
OpenDistro’s approach to index management doesn’t include a fixed number of states, but instead lets you define any number of them alongside different transitions. You can read more about it here.
Configuration of Index State Policies
In order to create an index state policy you can go to Index Management and select Create policy:
Then provide a policy ID name, and paste the following in the Define Policy section before clicking on Create:
{
"policy": {
"description": "Wazuh index state management for OpenDistro to move indices into a cold state after 30 days and delete them after a year.",
"default_state": "hot",
"states": [
{
"name": "hot",
"actions": [
{
"replica_count": {
"number_of_replicas": 1
}
}
],
"transitions": [
{
"state_name": "cold",
"conditions": {
"min_index_age": "30d"
}
}
]
},
{
"name": "cold",
"actions": [
{
"read_only": {}
}
],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_index_age": "365d"
}
}
]
},
{
"name": "delete",
"actions": [
{
"delete": {}
}
],
"transitions": []
}
],
"ism_template": {
"index_patterns": ["wazuh-alerts*"],
"priority": 100
}
}
}
The previous policy defines the following states:
Hot state. It sets 1 replica for the indices and a transition to the cold state when indices are older than 30 days.Cold state. It sets indices into read-only mode and a transition to the delete state when indices are older than 365 days.Delete state. Indices in this state are deleted.
For already existing indices you can use an API call or the Index Management tool.
Using the Wazuh Index Management tool
Go to the Index Management section and select Indices, search for the indices to which you wish to apply the policy, then click on Apply policy:
Now select the policy ID from the dropdown menu and click on Apply:
Using the OpenDistro API
Alternatively, you may use an API call to apply this setting to all wazuh alert indices. In order to do this, paste the following:
POST _opendistro/_ism/add/wazuh-alerts-3.x-*
{
"policy_id": "wazuh-index-state-policy"
}
into the Kibana Dev Tools console, and click on the triangle to send the request:
Conclusion
Ensuring that you only consume the storage resources that you need for your security platform is not only possible but easy to configure, which will help you maintain a healthy tool and in consequence guarantee you have visibility of your environment’s security standing.
References
Elastic ILM:
OpenDistro ISM:
If you have any questions about how to do the Wazuh index management, don’t hesitate to check out our documentation to learn more about Wazuh or join our community where our team and contributors will help you.