Creating decoders and rules from scratch

| by Juan Pablo Saez | Wazuh 3.12
Post icon

Wazuh provides an out-of-the-box set of rules used for threat detection and response. This ruleset is continuously updated thanks to the work of our contributors and developers. Moreover, users can easily create additional rules for ingesting and processing new log sources.

For this particular example, we will use local_decoder.xml and local_rules.xml files to implement the ruleset additions. These files are not overwritten when the ruleset gets updated, meaning that they are ideal for customizations and new decoders and rules. Besides, when users have to implement a large set of rules or decoders, they can create new decoder and rule files (this helps to keep them well organized).

Example event

When designing the new decoders and rules, you should consider event samples and taking a look at the manufacturer format reference. In this guide, we are working with the following Fortigate event:

date=2019-10-10 time=17:01:31 devname="FG111E-INFT2" devid="FG201E4Q17901611" logid="0000000020" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1573570891 srcip=192.168.56.105 srcname="wazuh.test.local" srcport=63874 srcintf="port1" srcintfrole="lan" dstip=54.97.146.111 dstport=443 dstintf="wan1" dstintfrole="wan" poluuid="3e421d8c-0210-51ea-2e5e-6dd151c37590" sessionid=261713795 proto=6 action="accept" user="WAZUH" authserver="FSSO_TEST_LOCAL" policyid=131 policytype="policy" service="HTTPS" dstcountry="United Kingdom" srccountry="Reserved" trandisp="snat" transip=195.46.111.2 transport=63874 appid=45553 app="Microsoft.Outlook.Office.365" appcat="Email" apprisk="medium" applist="INF-APP-MONITOR" appact="detected" duration=815 sentbyte=13941 rcvdbyte=13429 sentpkt=58 rcvdpkt=63 sentdelta=360 rcvddelta=2189 devtype="Windows PC" devcategory="Windows Device" osname="Windows" osversion="8.1" mastersrcmac="fc:45:96:44:79:c9" srcmac="fc:45:96:44:79:c9" srcserver=1 dstdevtype="Router/NAT Device" dstdevcategory="None" masterdstmac="28:8b:1c:db:7c:48" dstmac="28:8b:1c:db:7c:48" dstserver=0
  • At the syntax level, this log follows akey=value pairs structure. The pairs are whitespace-separated, and the string type values are surrounded by double-quotes.
  • As for the content, the log reveals network traffic between nodes and includes information as IP addresses, ports, protocol, and some payload details.

Decoder design

Services and devices log headers usually include static text meeting a pattern. It allows us to create a root decoder matching with all the logs sharing that particular pattern. The Regex (OS_Regex) syntax expressions are the tool we will use inside the decoders to easily locate the unchanging headers and their values.

It is good practice to first identify the log type in the prematch phase, and then use children decoder to extract the relevant data.

Decoder prematch

In the case of the Fortigate logs you can see the fields date, time and devname are always present at the beginning of every message, so they can be used in the root decoder prematch.

date=2019-10-10 time=17:01:31 devname="FG111E-INFT2"

The following decoder will attempt to find a match on every log for the expression defined and decide whether the child decoders should be considered for triggering or not.

<decoder name="fortigate-custom">
  <prematch>^date=\d\d\d\d-\d\d-\d\d time=\d\d:\d\d:\d\d devname="\S+"</prematch>
</decoder>
  • The field date will always contain the Year–month–day pattern. Year, month and day are always digits. The \d operator for 0-9 characters fits for the date field.
  • The field time has the Hour:minute:second format. Hour, minute and seconds are always be digits. The \d operator for 0-9 characters fits for the time field.
  • The field devname is expected to be under double quotes and not contain whitespaces. The anything but spaces operator \S is a good way to reference the devname value.

Decoder field extraction

The decoder below extracts the values for the headers date, time and devname. The regex option finds the fields of interest and extracts them through the () operator. The order option defines what the parenthesis groups contain and the order in which they were received.

<decoder name="fortigate-custom1">
  <parent>fortigate-custom</parent>
  <regex>^date=(\d\d\d\d-\d\d-\d\d) time=(\d\d:\d\d:\d\d) devname="(\S+)"</regex>
  <order>date, time, devname</order>
</decoder>

The following decoder extracts the devid, login, type and subtype fields values, with the same logic as the one above.

<decoder name="fortigate-custom1">
  <parent>fortigate-custom</parent>
  <regex>devid="(\S+)" logid="(\S+)" type="(\S+) subtype="(\S+)"</regex>
  <order>devid, logid, type, subtype</order>
</decoder>

The devid, logid,type and subtype fields are extracted using \S operator. It matches anything but whitespaces. To find fields containing whitespaces,  the \. operator should be used.

Extracting more fields

When dealing with dynamically structured logs that provide information in various orders or sometimes omitting certain values, using modular logic decoders as the ones below could be game-changing. Check the “Sibling decoders: flexible extraction of information” blogpost for further information on this.

<decoder name="fortigate-custom1">
  <parent>fortigate-custom</parent>
  <regex>srcip="(\S+)"|srcip=(\S+) </regex>
  <order>srcip</order>
</decoder>

<decoder name="fortigate-custom1">
  <parent>fortigate-custom</parent>
  <regex>dstip=(\S+) </regex>
  <order>dstip</order>
</decoder>

<decoder name="fortigate-custom1">
  <parent>fortigate-custom</parent>
  <regex>action="(\S+)" </regex>
  <order>action</order>
</decoder>

By using the | operator, the first decoder above covers two possible syntax patterns for the srcip field: one with the address under double-quotes and the other without them.

Rules design

Rules perform checks over the extracted fields to identify the type of message received. The final rule matched will determine whether an alert is generated or not, its level, and category groups. The following grouping rule will generate an alert for every event that triggers the Fortigate decoders:

<rule id="222000" level="3">
  <decoded_as>fortigate-custom</decoded_as>
  <description>Fortigate messages grouped.</description>
</rule>

Pointing the grouping rule above with the <if_sid> option, any number of children rules could be created. These rules could check for values giving a hint about an attack or a behavior you want to be alerted to. The following sample rule checks for a failed login on the action and status fields.

<rule id="222016" level="4">
  <if_sid>222000</if_sid>
  <action>login</action>
  <status>failed</status>
  <description>Fortigate: Login failed.</description>
</rule>

Testing the decoders and rules

Once the decoders and rules have been designed, it is necessary to ensure that they work properly. It should be verified using the ossec-logtest tool and also by ingesting the logs through the normal flow to see if they generate an alert.

ossec-logtest

The following is the output after entering the example log on ossec-logtest.

**Phase 2: Completed decoding.
       decoder: 'fortigate-custom'
       date: '2019-10-10'
       time: '17:01:31'
       devname: 'FG111E-INFT2'
       devid: 'FG201E4Q17901611'
       logid: '0000000020'
       type: 'traffic"'
       subtype: 'forward'
       srcip: '192.168.56.105'
       dstip: '54.97.146.111'
       action: 'accept'
**Phase 3: Completed filtering (rules). Rule id: '222000' Level: '3' Description: 'Fortigate messages grouped.' **Alert to be generated.

Ingesting the sample event

For this test, we are creating a new dummy log: /var/log/test_file.log.

$ touch /var/log/test_file.log

Then we should set Wazuh to monitor this log file. The following configuration block should be pasted on the Wazuh manager ossec.conffile. Remember to restart the manager after adding this setting:

<localfile>
  <log_format>syslog</log_format>
  <location>/var/log/test_file.log</location>
</localfile>

Time to throw the sample event into /var/log/test_file.log. If our decoders and rules are working properly, the alert regarding the event should appear on the alerts.log file.

echo "date=2019-10-10 time=17:01:31 devname="FG111E-INFT2" devid="FG201E4Q17901611" logid="0000000020" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1573570891 srcip=192.168.56.105 srcname="wazuh.test.local" srcport=63874 srcintf="port1" srcintfrole="lan" dstip=54.97.146.111 dstport=443 dstintf="wan1" dstintfrole="wan" poluuid="3e421d8c-0210-51ea-2e5e-6dd151c37590" sessionid=261713795 proto=6 action="accept" user="WAZUH" authserver="FSSO_TEST_LOCAL" policyid=131 policytype="policy" service="HTTPS" dstcountry="United Kingdom" srccountry="Reserved" trandisp="snat" transip=195.46.111.2 transport=63874 appid=45553 app="Microsoft.Outlook.Office.365" appcat="Email" apprisk="medium" applist="INF-APP-MONITOR" appact="detected" duration=815 sentbyte=13941 rcvdbyte=13429 sentpkt=58 rcvdpkt=63 sentdelta=360 rcvddelta=2189 devtype="Windows PC" devcategory="Windows Device" osname="Windows" osversion="8.1" mastersrcmac="fc:45:96:44:79:c9" srcmac="fc:45:96:44:79:c9" srcserver=1 dstdevtype="Router/NAT Device" dstdevcategory="None" masterdstmac="28:8b:1c:db:7c:48" dstmac="28:8b:1c:db:7c:48" dstserver=0" >> /var/log/test_file.log

The alert above appears on the alerts.log file:

** Alert 1587490736.163684: - local,syslog,sshd,
2020 Apr 21 17:38:56 serv-test-manager-centos-1->/var/log/test_file.log
Rule: 222000 (level 3) -> 'Fortigate messages grouped.'
Src IP: 192.168.56.105
Dst IP: 54.97.146.111
date=2019-10-10 time=17:01:31 devname="FG111E-INFT2" devid="FG201E4Q17901611" logid="0000000020" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1573570891 srcip=192.168.56.105 srcname="wazuh.test.local" srcport=63874 srcintf="port1" srcintfrole="lan" dstip=54.97.146.111 dstport=443 dstintf="wan1" dstintfrole="wan" poluuid="3e421d8c-0210-51ea-2e5e-6dd151c37590" sessionid=261713795 proto=6 action="accept" user="WAZUH" authserver="FSSO_TEST_LOCAL" policyid=131 policytype="policy" service="HTTPS" dstcountry="United Kingdom" srccountry="Reserved" trandisp="snat" transip=195.46.111.2 transport=63874 appid=45553 app="Microsoft.Outlook.Office.365" appcat="Email" apprisk="medium" applist="INF-APP-MONITOR" appact="detected" duration=815 sentbyte=13941 rcvdbyte=13429 sentpkt=58 rcvdpkt=63 sentdelta=360 rcvddelta=2189 devtype="Windows PC" devcategory="Windows Device" osname="Windows" osversion="8.1" mastersrcmac="fc:45:96:44:79:c9" srcmac="fc:45:96:44:79:c9" srcserver=1 dstdevtype="Router/NAT Device" dstdevcategory="None" masterdstmac="28:8b:1c:db:7c:48" dstmac="28:8b:1c:db:7c:48" dstserver=0"
date: 2019-10-10
time: 17:01:31
devname: FG111E-INFT2
devid: FG201E4Q17901611
logid: 0000000020
type: traffic"
subtype: forward

References

If you have any questions about this, don’t hesitate to check out our documentation to learn more about Wazuh or join our community where our team and contributors will help you.