Why does Wazuh need an Elasticsearch template?

| by Jesús González | Wazuh 3.7
Post icon

On the Elasticsearch website, we can find the following description:

Index templates allow you to define templates that will automatically be applied when new indices are created. The templates include settings, mappings and a simple pattern template that controls whether the template should be applied to the new index.

Elastic and Wazuh logos

Elasticsearch doesn’t know the desired structure and format for our data. It receives a lot of alerts and tries to parse them, that’s all. If no template is available, it creates an index using what it thinks is correct, but it’s wrong of course. For example, it splits depending on spaces included in a string, and that’s not the desired behavior for us.

Let’s assume we have the following fields:

"data count" - 12 occurrences
"data releases" - 2 occurrences
"data patches" - 1 occurrences

What we’d get if we don’t use a template is 15 hits for data: 12 hits for count, 2 hits for releases and 1 hit for patches, and that’s not the desired result for us. Instead, we usually have to set types for certain fields. That means we must tell Elasticsearch how to use certain fields (IP, string, number, object, etc.). If we don’t set the fields properly, an unexpected behavior will occur in the Kibana dashboards, the Wazuh app, and a few other known integrations. The template will also reject incorrectly formatted events. If a corrupted alert is sent to Elasticsearch, but our template doesn’t match with its format, it will be rejected.

More benefits of using a template

The template defines the index pattern to be applied. An index pattern in Elasticsearch means a regular expression that matches with a range of indices. Let’s look at the Logstash configuration output section:

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "wazuh-alerts-3.x-%{+YYYY.MM.dd}"
    document_type => "wazuh"
  }
}

It says we want to send data to be indexed in a wazuh-alerts-3.x-%{+YYYY.MM.dd} index, such as wazuh-alerts-3.x-2018.09.13. Now let’s see how our template begins:

{
 "wazuh" : {
   "order" : 0,
   "index_patterns" : [
     "wazuh-alerts-3.x-*"
   ],

The index pattern wazuh-alerts-3.x-* matches with wazuh-alerts-3.x-2018.09.13, so this template should be applied to this index.

Note: When we talk about an Elasticsearch index pattern, we are not talking about a Kibana index pattern*.

Elasticsearch pattern. Diagram.

Some useful commands regarding Wazuh and Elasticsearch templates

Check the existence of the Wazuh template:

curl localhost:9200/_cat/templates/wazuh

Check content from our template:

curl localhost:9200/_template/wazuh

Check which template is being applied for a specific index:

curl localhost:9200/wazuh-alerts-3.x-2018.09.13/_mapping?pretty

What happens if an index is created before inserting the template?

What you have is a corrupted index that is incompatible with our current integration. The only way to solve this problem is to re-map your index. This means you should create a new index using the data from the corrupted index and apply the template.  Once you are done, you can replace the corrupted index with the well-mapped index.

Be aware of Logstash priority

Even if you have inserted the template properly, Logstash tries to create indices by sending data to Elasticsearch. If Logstash sent data before the template was inserted, then Elasticsearch creates the index without using the template. Since an index mapping can’t be modified once it’s created, that index won’t use your template.

If you have any questions about this, don’t hesitate to check out our documentation to learn more about Wazuh or join our community where our team and contributors will help you.