Wazuh helps you comply with the security standards in which logs are required to be maintained for several months so that they can be provided on the spot in case of an audit.

Being able to quickly access all this information requires storing it on hard disks. Since storage space has a cost and a limit, you may have to delete old data to ensure you can maintain the retention period that you need.

Alerts generated by Wazuh are sent to an Elasticsearch daily index named wazuh-alerts-3.x-YYYY.MM.DD by using the default configuration.

In this blog post you will learn how to configure Elastic ILM and OpenDistro ISM to automatically manage the data in those indices over time.

Elastic ILM

You can create policies that govern what is the lifecycle of the indices based on different phases.

Four phases can be defined in a Lifecycle Policy:

  • Hot phase. For recent data that is actively accessed.
  • Warm phase. Data that you may wish to access, but less often.
  • Cold phase. Similar to the warm phase but you may also freeze indices to reduce overhead.
  • Delete phase. Data that reaches this phase is deleted.

Create a policy

To configure an index lifecycle policy you may go into the Management section of your Kibana web interface, select Index Lifecycle Policies and then click on the Create policy button:
Navigation to the Policy Creation tool. First select the settings menu on the bottom left, then Index Lifecycle Policies and finally on the far right click on Create Policy.

Once there you can specify the various options of the policy:

Create an index lifecycle policy dialog. This dialog contains all the options that will define the policy.

  1. Provide a name to the policy.
  2. You may disable the rollover option unless you wish to use it. For more information see using rollover policies.
  3. You can reduce the overhead of an index by enabling a cold phase.
  4. Specify a time after the cold phase will be applied.
  5. Activate the delete phase.
  6. Specify the age of the index before it is deleted.
  7. Finally, save your new policy.

Note: If you have elasticsearch nodes that have hardware of lower performance and cost, you may specify that during the cold phase the data will be stored on these nodes. For more information see shard allocation.

Add a policy to an index template

The next step is to apply this new policy to the index template of Wazuh alerts. In order to do so unfurl the Actions menu and select Add policy to index template:
Applying the policy to an index template. To the right of the newly created lifecycle policy select Actions and then click on Add policy to index template.

Then select wazuh from the index template drop-down menu, and click on Add policy:
Adding the policy to the wazuh template. Select wazuh from the dropdown Index template menu and then click on Add Policy

This will apply the policy to all wazuh alerts indices created in the future.

To apply this to already existing indices you can use an API call or the Index Management tool.

Using the Index Management tool

You may search and select the indices to which you wish to apply the policy, then select Add lifecycle policy from the Manage index menu:
Selecting existing indices to add lifecycle policy

Once there select the newly created policy and click on Add policy:
Selecting which policy to apply to the selected index.

Using the Elasticsearch API

Alternatively, you may use an API call to apply this setting to all wazuh alert indices. Paste the following:

PUT wazuh-alerts-3.x-*/_settings 
{
  "index.lifecycle.name":"wazuh-alert-retention-policy"
}

into the Kibana Dev Tools console, and click on the triangle to send the request:
Elasticsearch Dev Tools Console. Reach the console by clicking on the wrench icon on the left, then paste the code and click on the arrow to submit the request.

The system will reply with {"acknowledged" : true}.

With this, the Elastic ILM configuration is completed.

OpenDistro ISM

Open Distro is a fully open-source project maintained by Amazon Web Services which aims to provide an alternative to the proprietary features of Elasticsearch. Although very similar, there are some key differences, one of which is how to automatically manage the lifecycle of indices.

OpenDistro’s approach to index management doesn’t include a fixed number of states, but instead lets you define any number of them alongside different transitions. You can read more about it here.

Configuration of Index State Policies

In order to create an index state policy you can go to Index Management and select Create policy:
To create a policy select the IM icon from the icon column on the left and then click the Create Policy button in the middle. Provided you are in the Index Policies section.

Then provide a policy ID name, and paste the following in the Define Policy section before clicking on Create:

{
    "policy": {
        "description": "Wazuh index state management for OpenDistro to move indices into a cold state after 30 days and delete them after a year.",
        "default_state": "hot",
        "states": [
            {
                "name": "hot",
                "actions": [
                    {
                        "replica_count": {
                            "number_of_replicas": 1
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "cold",
                        "conditions": {
                            "min_index_age": "30d"
                        }
                    }
                ]
            },
            {
                "name": "cold",
                "actions": [
                    {
                        "read_only": {}
                    }
                ],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "365d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ]
    }
}

To create a policy first provide a name for it on the Policy ID box and then place the code for the policy inside the Define Policy box. Finally click on the Create button on the bottom left.

The previous policy defines the following states:

  • Hot state. It sets 1 replica for the indices and a transition to the cold state when indices are older than 30 days.
  • Cold state. It sets indices into read-only mode and a transition to the delete state when indices are older than 365 days.
  • Delete state. Indices in this state are deleted.

For this to be applied to indices created in the future you must change the index template to include this policy. Run the following commands on the Wazuh Manager’s command line:

sed -i 's/  "settings": {/  "settings": {\n    "opendistro.index_state_management.policy_id": "wazuh-index-state-policy",/g' /etc/filebeat/wazuh-template.json
filebeat setup --index-management

For already existing indices you can use an API call or the Index Management tool.

Using the Index Management tool

Go to the Index Management section and select Indices, search for the indices to which you wish to apply the policy, then click on Apply policy:
In the indices section of the Index Management utility, you may use the search bar to find the existing wazuh-alerts indices, select them all and then click on Apply Policy

Now select the policy ID from the dropdown menu and click on Apply:
Apply policy dialog. Here you can select the Policy ID and click on Apply to apply the policy to the previously selected indices

Using the OpenDistro API

Alternatively, you may use an API call to apply this setting to all wazuh alert indices. In order to do this, paste the following:

POST _opendistro/_ism/add/wazuh-alerts-3.x-*
{
  "policy_id": "wazuh-index-state-policy"
}

into the Kibana Dev Tools console, and click on the triangle to send the request:
Selecting the Dev Tools plugin by clicking on the wrench icon on the left column, then pasting the code presented above and clicking on the Play arrow will submit the API request to apply the indices.

Conclusion

Ensuring that you only consume the storage resources that you need for your security platform is not only possible but easy to configure, which will help you maintain a healthy tool and in consequence guarantee you have visibility of your environment’s security standing.

References

Elastic ILM:

OpenDistro ISM: