Threat hunting is a proactive cybersecurity approach focused on actively searching for indicators of compromise (IOCs) or indicators of attack (IOAs) within an organization’s IT environment. To maintain a good security posture, organizations must continually monitor network and system activities to find hints and digital footprints that could be IOCs and IOAs. These IOCs or IOAs can be found in inventory data, helping organizations identify potential security threats or anomalies in monitored endpoints.

After installing a Wazuh agent on an endpoint, the Wazuh Syscollector module gathers relevant information for system inventory. This module scans monitored endpoints for data covering hardware, operating system, network interfaces, packages, ports, and processes. Each scan retrieves a list of fields containing the collected data, which is then stored in a database for further analysis.

Users can generate system inventory reports from the Wazuh dashboard, which can be used for hunting anomalies,  intruders in their networks, undesired software, or incorrect parameters on a process. In this blog post, we explore practical applications of threat hunting with the system inventory data collected by the Wazuh Syscollector module. Additionally, we explore how to leverage the data to trigger alerts, and how to use Wazuh Query Language (WQL) within the inventory data.

How to use decoded fields in the Wazuh ruleset

You can create rules to trigger alerts based on the system inventory information collected by the Wazuh Syscollector module. The Wazuh analysis engine groups Syscollector events with the rule ID 221 and the rule level is 0, so by default, it is not shown on the Wazuh dashboard. To visualize Syscollector events on the Wazuh dashboard, you must create custom rules that inherit the parent rule ID 221 with a severity level of 3 or higher. For example, the custom rule below is triggered when a logical port is opened on a monitored endpoint:

<group name="syscollector,">
  <!-- ports -->
  <rule id="100310" level="3">
    <if_sid>221</if_sid>
    <field name="type">dbsync_ports</field>
    <description>Syscollector ports event.</description>
  </rule>

  <rule id="100311" level="3">
    <if_sid>100310</if_sid>
    <field name="operation_type">INSERTED</field>
    <description>The port: $(port.local_port), with local ip: $(port.local_ip) has been opened. Syscollector creation event detected.</description>
  </rule>

</group>

With the new rule added, an alert is displayed each time a port is opened which can be viewed on the Wazuh dashboard. By displaying alerts based on collected system information, it is easier to detect threats.

Detect threats

Note: The alert is triggered after the second Syscollector scan when an information difference is detected. This second scan occurs when the configured interval is reached.

To aid the creation of rules in the threat hunting process, you can search for any Syscollector field on the Wazuh dashboard. The Wazuh indexer saves the Syscollector fields as data.type.value. For example, in hardware type, the cpu_name field is data.hardware.cpu_name. This can be seen below when investigating logs collected on the Wazuh dashboard.

Threat Hunting process

Infrastructure

To demonstrate threat hunting using inventory data in Wazuh, we set up the following infrastructure.

  • A pre-built, ready-to-use Wazuh OVA 4.7.4 that includes the Wazuh central components (Wazuh server, Wazuh indexer, and Wazuh dashboard). Follow this guide to download the virtual machine (VM).
  • An Ubuntu 22.04 endpoint with Wazuh agent 4.7.4 installed and enrolled to the Wazuh server.

Use cases

We highlight use cases for threat hunting using inventory data on Wazuh. We implement CDB lists and custom rules to detect the following use cases.

We also use the Wazuh Query Language (WQL) to hunt for the following use cases.

Prerequisite

On the Ubuntu endpoint, the Wazuh Syscollector module is enabled by default and set with a 1h scan interval. Configure the Wazuh agent to set a minimal interval for syscollector scans and ensure that inventory data is rapidly synchronized with the Wazuh dashboard. For the purpose of this demonstration, we use an interval of 1m. The port scan of the Syscollector module is also set to detect all the established connections.

1. Edit the syscollector block in the Wazuh agent configuration file /var/ossec/etc/ossec.conf to use a 1m interval and set ports all to yes as shown below:

<!-- System inventory -->
<wodle name="syscollector">
  <disabled>no</disabled>
  <interval>1m</interval>
  <scan_on_start>yes</scan_on_start>
  <hardware>yes</hardware>
  <os>yes</os>
  <network>yes</network>
  <packages>yes</packages>
  <ports all="yes">yes</ports>
  <processes>yes</processes>

2. Restart the Wazuh agent for the configuration changes to take effect:

# systemctl restart wazuh-agent

Detecting unknown ports

Detecting abnormal ports is a fundamental aspect of proactive threat hunting using system inventory data. By analyzing network traffic collected by the Wazuh Syscollector scans, organizations can uncover irregularities that may signal unauthorized access attempts or malicious activities.

Perform the actions below to threat hunt for potentially unauthorized or malicious network activities involving ports outside of the specified authorized ports.

Wazuh server

Configure the Wazuh server to expect a list of ports that we do not want to alert on by creating a CDB list and writing rules to detect connections from unwanted ports and IP addresses connecting through the port.

1. Create a CDB list of known common ports /var/ossec/etc/lists/common-ports:

# touch /var/ossec/etc/lists/common-ports

2. Add the list of common ports that we do not want to alert on:

25:
22:
53:
80:
135:
389:
443:
445:
993:
995:
1514:
1515:
3389:
5000:
5223:
8000:
8002:
8080:
8083:
8443:
9000:

3. Change the ownership and permissions of the list created:

# chown wazuh:wazuh /var/ossec/etc/lists/common-ports
# chmod 660 /var/ossec/etc/lists/common-ports

4. Edit the Wazuh server configuration file /var/ossec/etc/ossec.conf and add the CDB list created within the <ruleset> block:

  <ruleset>
    <!-- Default ruleset -->
    <decoder_dir>ruleset/decoders</decoder_dir>
    <rule_dir>ruleset/rules</rule_dir>
    <rule_exclude>0215-policy_rules.xml</rule_exclude>
    <list>etc/lists/audit-keys</list>
    <list>etc/lists/amazon/aws-eventnames</list>
    <list>etc/lists/security-eventchannel</list>
    <list>etc/lists/common-ports</list>

    <!-- User-defined ruleset -->
    <decoder_dir>etc/decoders</decoder_dir>
    <rule_dir>etc/rules</rule_dir>
  </ruleset>

5. Add the rules below to the /var/ossec/etc/rules/local_rules.xml file to generate alerts when a specific port is opened :

<group name="syscollector,">
  <!-- ports -->
  <rule id="100300" level="2">
      <if_sid>221</if_sid>
      <field name="type">dbsync_ports</field>
      <description>Syscollector ports event.</description>
  </rule>

<!-- Abnormal Destination Port Detected -->
  <rule id="100370" level="9">
    <if_sid>100300</if_sid>
    <field name="operation_type">INSERTED</field>
    <field name="port.process" negate="yes">wazuh-agentd</field>
    <list field="port.local_port" lookup="not_match_key">etc/lists/common-ports</list>
    <description>Network connection to Uncommon Port $(port.local_port) from $(port.remote_ip)</description>
  </rule>

</group>

6. Restart the Wazuh manager service to implement the changes:

# systemctl restart wazuh-manager

Simulating the test

Simulate a test on the Ubuntu endpoint to detect unknown port 21 that was not expected and included in our CDB list.

Install the vsftpd package, it opens port 21 to listen on:

$ sudo apt install vsftpd

Visualizing the alert

Perform the following steps to view the alerts on the Wazuh dashboard.

1. Navigate to Modules > Security Events > Events.

2. Click + Add filter. Then filter for rule.id in the Field field.

3. Filter for is in the Operator field.

4. Filter for 100370 in the Values field.

5. Click Save.

Wazuh dashboard alerts

Clean up commands

Execute the following commands to uninstall the package and delete all related files to the package.

Uninstall the vsftpd package:

$ sudo apt autoremove vsftpd
$ sudo apt purge vsftpd

Detecting a new network interface

Maintaining a clear view and management of network configurations is vital for upholding system security and integrity. Attackers often add new network interfaces to compromised systems as a tactic to gain unauthorized network access, extract data, or execute malicious activities. Detecting these changes promptly is crucial to stopping potential security threats and preventing further compromise.

Perform the action below to generate an alert when a new network interface is added to a monitored endpoint.

Wazuh server

1. Add the rules below to the /var/ossec/etc/rules/local_rules.xml file to monitor and alert for new network interfaces detected by the Wazuh syscollector:

<group name="syscollector2,">
  
<!-- New network interface -->
  <rule id="100371" level="2">
    <if_sid>221</if_sid>
    <field name="type">dbsync_network_iface</field>
    <description>Syscollector network interface event.</description>
  </rule>

 <rule id="100372" level="9">
   <if_sid>100371</if_sid>
   <field name="operation_type">INSERTED</field>
   <description>New network interface $(netinfo.iface.name) detected.</description>
  </rule>

</group>

2. Restart the Wazuh manager service to implement the changes:

# systemctl restart wazuh-manager

Simulating the test

Simulate the test on the Ubuntu endpoint to detect a new network interface to test our rules created from the collected system inventory information.

1. Create a new virtual ethernet interface eth1 and eth-peer:

$ sudo ip link add eth1 type veth peer name eth1-peer

2. Bring up the new interface eth1:

$ sudo ip link set dev eth1 up

Visualizing the alert

Perform the following steps to view the alerts on the Wazuh dashboard.

1. Navigate to Modules > Security Events > Events.

2. Click + Add filter. Then filter for rule.id in the Field field.

3. Filter for is in the Operator field.

4. Filter for 100372 in the Values field.

5. Click Save.

Threat hunting alerts

Clean up commands

Execute the following commands to delete the newly created virtual network interface.

1. Bring down the new interface eth1:

$ sudo ip link set dev eth1 down

2. Delete the interfaces eth1 and eth-peer:

$ sudo ip link delete eth1

Using Wazuh Query Language in inventory data

Wazuh Query Language (WQL) aids users in sifting through vast datasets, facilitating data filtering within the system inventory. WQL enhances search bar functionality across tabs like Security Configuration Assessment, rule management, and more within the Wazuh dashboard. This data filtering capability enhances users’ ability to identify and respond to potential security threats effectively. 

You can view the system inventory data of the monitored endpoint on the Wazuh dashboard. 

1. Navigate to the Agents section on your Wazuh dashboard.

2. Select the monitored endpoint.

3. Navigate to Inventory data.

The inventory data page for each monitored endpoint shows its operating system, hardware, processes, network interface, and packages. The Wazuh Query Language (WQL) expands the search bar functionality of each tab of the inventory data on the Wazuh dashboard.

Wazuh Query Language

WQL allows you to efficiently query available inventory fields like ram_usage, name, and version of packages installed, cmd commands executed from processes running, and many more. This functionality enables threat hunters to gain valuable insights into unwanted packages and processes, and abnormal resource consumption.

Some examples of using the WQL in inventory data for threat hunting are highlighted below:

Outdated packages installed

Actively seeking out outdated packages is crucial for threat hunting, as they frequently harbor known vulnerabilities exploitable by attackers. We will look for an OpenSSL package installed on an endpoint with a version less than 3.0.8. For versions of OpenSSL lower than 3.0.8, one notable vulnerability is CVE-2024-2511, causing unbounded memory growth in TLSv1.3 sessions. 

Run the query below on the WQL search bar in the Packages section:

name ~ openssl and version < 3.0.8

Output:

An OpenSSL package lower than version 3.0.8 is visible as a result.

OpenSSL package

Insecure processes running

Monitoring processes running as the root user but not in a sleeping state could help identify a potentially insecure activity. Such processes often have elevated privileges and are actively executing commands on the system, raising concerns for potential exploitation or unauthorized access.

Run the query below in the WQL search bar in the Processes section:

euser="root" and state!= "S"

Output:

Processes running as the root user and not in a sleeping state are displayed.

Threat Hunting

Conclusion

Inventory data collected by the Wazuh Syscollector module can significantly aid threat hunting endeavors. Information gathered can be used to create rules and logic for the detection of undesired software, threats, errors in network interfaces, and other anomalous activities. Through our exploration, we’ve demonstrated how to trigger alerts using system inventory data, utilize the Wazuh Query Language (WQL) within this dataset, and effectively conduct threat hunting operations, underscoring its pivotal role in fortifying our security posture.

Don’t hesitate to ask any questions about how to detect threats and intruders or share your use cases for this feature in our social media, mailing list, or Slack community.