Detecting and responding to malicious files using CDB lists and active response

Malicious files can serve as indicators of compromise (IOC) on endpoints where they are observed to be present. These files may end up on endpoints through various attack vectors. As such, it is important to detect and respond to them as soon as they are downloaded. Wazuh has a file integrity monitoring (FIM) component that detects and alerts when files are created, modified, or deleted. The alerts generated by the FIM component contain the file MD5, SHA1, and SHA256 checksums in their metadata.

In our proof of concept guide, we show how to detect and respond to malicious files using VirusTotal and Yara. In this post, we look at detecting malicious files using their MD5 checksums and a constant database (CDB) list of known malicious MD5 hashes. If a file hash is present in the CDB list, a file delete action is taken on it using the Wazuh active response module.

Configuring directory monitoring

On the agent, we can monitor a directory for file changes by adding the <directories> block specifying the folder to be monitored in the agent configuration file. Then we restart the agent.

<ossec_config>
  <syscheck>
    <disabled>no</disabled>
    <directories check_all="yes" realtime="yes">MONITORED_DIRECTORY_PATH</directories>
  </syscheck>
</ossec_config>

check_all allows checks of the file size, permissions, owner, last modification date, inode, and the hash sums (MD5, SHA1, and SHA256).

When we proceed to download a file to the directory configured for FIM checks, we get an alert with rule ID 554 that warns that a file has been added to the system.

Creating the CDB list

A CDB list is a text file with key:value pairs. Each pair must be on a single line, and the keys must be unique. However, values are optional. In this post, we use a CDB list to create a malware blacklist containing MD5 hashes of known malicious files.

To do this, create a file called malware-hashes in /var/ossec/etc/lists/ on the manager.

touch /var/ossec/etc/lists/malware-hashes

A sample entry in the file with the hash for the eicar malware test file is:

44d88612fea8a8f36de82e1278abb02f:

More MD5 malware hashes can be gotten from VirusShare or Virustotal. The Python script below formats a hash list file downloaded from VirusShare for use in a CDB list.

with open('<path_to_downloaded_hashes_list>', 'r') as istr:
    with open('malware-hashes', 'w') as ostr:
        for i, line in enumerate(istr):
            if i > 5:
                line = line.rstrip('\n')
                line += ':'
                print(line, file=ostr)

We proceed to add the created CDB list to the manager ossec.conf so it is available for use in rules. The list is added to the manager by specifying the path to the list in the <ruleset> block.

<list>etc/lists/malware-hashes</list>  

Detecting Malicious Files

Once the list has been added to the configuration file, we proceed to create a custom rule in /var/ossec/etc/rules/local_rules.xml to alert when the hash of a downloaded file is found in the malware blacklist.

<group name="local,malware,">
  <rule id="100002" level="5">
    <if_sid>554</if_sid>
    <list field="md5" lookup="match_key">etc/lists/malware-hashes</list>
    <description>A file - $(file) - in the malware blacklist was added to the system.</description>
  </rule>

  <rule id="100003" level="5">
    <if_sid>100002</if_sid>
    <field name="file" type="pcre2">(?i)[c-z]:</field>
    <description>A file - $(file) - in the malware blacklist was added to the system.</description>
  </rule>
</group>

Then, we restart the manager to apply the rule and configuration changes.

systemctl restart wazuh-manager

To test the detection rule, we proceed to download a malicious file (in this case, we are using the eicar.com test file available here) to the directory being monitored by the FIM module. We see an alert indicating that the downloaded file hash is in the malware blacklist.

Configuring active response

Active response allows Wazuh to run commands on an endpoint in response to certain triggers. For example, if a specific rule is triggered and it generates an alert, an active response command can be run on the endpoint that generated that alert. In this case, we would like to run a Python script on a Windows agent to remove the malicious file downloaded.

It is important to note the following items in the active response script:

• The active response script is created on the agent.
• If the active response script is being run on Linux-based endpoints, the first line on the script must indicate the Python interpreter.

#!/usr/bin/python3

• The os.remove() function is the function that handles the removal of the malicious file.

os.remove(msg.alert["parameters"]["alert"]["syscheck"]["path"])

• The outcome of the file removal action is logged to the file active-responses.log. Its path is dependent on the operating system the script is run on. In our case, it is C:\Program Files(x86)\ossec-agent\active-response\active-responses.log

The file remove-threat.py is created with the contents below.

#!/usr/bin/python3
# Copyright (C) 2015-2022, Wazuh Inc.
# All rights reserved.
 
import os
import sys
import json
import datetime
 
if os.name == 'nt':
    LOG_FILE = "C:\Program Files (x86)\ossec-agent\active-response\active-responses.log"
else:
    LOG_FILE = "/var/ossec/logs/active-responses.log"
 
ADD_COMMAND = 0
DELETE_COMMAND = 1
CONTINUE_COMMAND = 2
ABORT_COMMAND = 3
 
OS_SUCCESS = 0
OS_INVALID = -1
 
class message:
    def __init__(self):
        self.alert = ""
        self.command = 0
 
def write_debug_file(ar_name, msg):
    with open(LOG_FILE, mode="a") as log_file:
        log_file.write(str(datetime.datetime.now().strftime('%Y/%m/%d %H:%M:%S')) + " " + ar_name + ": " + msg +"\n")
 
def setup_and_check_message(argv):
 
    # get alert from stdin
    input_str = ""
    for line in sys.stdin:
        input_str = line
        break
 
 
    try:
        data = json.loads(input_str)
    except ValueError:
        write_debug_file(argv[0], 'Decoding JSON has failed, invalid input format')
        message.command = OS_INVALID
        return message
 
    message.alert = data
 
    command = data.get("command")
 
    if command == "add":
        message.command = ADD_COMMAND
    elif command == "delete":
        message.command = DELETE_COMMAND
    else:
        message.command = OS_INVALID
        write_debug_file(argv[0], 'Not valid command: ' + command)
 
    return message
 
 
def send_keys_and_check_message(argv, keys):
 
    # build and send message with keys
    keys_msg = json.dumps({"version": 1,"origin":{"name": argv[0],"module":"active-response"},"command":"check_keys","parameters":{"keys":keys}})
 
    write_debug_file(argv[0], keys_msg)
 
    print(keys_msg)
    sys.stdout.flush()
 
    # read the response of previous message
    input_str = ""
    while True:
        line = sys.stdin.readline()
        if line:
            input_str = line
            break
 
    # write_debug_file(argv[0], input_str)
 
    try:
        data = json.loads(input_str)
    except ValueError:
        write_debug_file(argv[0], 'Decoding JSON has failed, invalid input format')
        return message
 
    action = data.get("command")
 
    if "continue" == action:
        ret = CONTINUE_COMMAND
    elif "abort" == action:
        ret = ABORT_COMMAND
    else:
        ret = OS_INVALID
        write_debug_file(argv[0], "Invalid value of 'command'")
 
    return ret
 
def main(argv):
 
    write_debug_file(argv[0], "Started")
 
    # validate json and get command
    msg = setup_and_check_message(argv)
 
    if msg.command < 0:
        sys.exit(OS_INVALID)
 
    if msg.command == ADD_COMMAND:
        alert = msg.alert["parameters"]["alert"]
        keys = [alert["rule"]["id"]]
        action = send_keys_and_check_message(argv, keys)
 
        # if necessary, abort execution
        if action != CONTINUE_COMMAND:
 
            if action == ABORT_COMMAND:
                write_debug_file(argv[0], "Aborted")
                sys.exit(OS_SUCCESS)
            else:
                write_debug_file(argv[0], "Invalid command")
                sys.exit(OS_INVALID)
 
        try:
            os.remove(msg.alert["parameters"]["alert"]["syscheck"]["path"])
            write_debug_file(argv[0], json.dumps(msg.alert) + " Successfully removed threat")
        except OSError as error:
            write_debug_file(argv[0], json.dumps(msg.alert) + "Error removing threat")
           
       
    else:
        write_debug_file(argv[0], "Invalid command")
 
    write_debug_file(argv[0], "Ended")
 
    sys.exit(OS_SUCCESS)
 
if __name__ == "__main__":
    main(sys.argv)

For Windows endpoints

Since we are running the active response script on a Windows agent, we perform the following tasks:

• Create an executable from the Python script so it can run on all Windows agents regardless of if they have python installed or not. This executable can be created by using pyinstaller to convert the script to an executable file.

pyinstaller -F remove-threat.py

• We copy the built executable to C:\Program Files (x86)\ossec-agent\active-response\bin.

For Linux based endpoints

If the active response script is being run on a Linux based endpoint, the following steps should be taken:

• Place the Python script created in the active response directory.

cp remove-threat.py /var/ossec/active-response/bin/remove-threat

• Change the file owner and group to root:ossec, then give the script execution permissions.

chmod 750 /var/ossec/active-response/bin/remove-threat
chown root:ossec /var/ossec/active-response/bin/remove-threat

• The active response <rules_id> will be 100002.

Configuration on the Manager

Now that the active response executable has been placed in the bin folder on the agent, we proceed to configure the manager to trigger an active response when the malware blacklist detection rule is triggered. In the manager configuration file, we add the following block in theossec_config block:

<command>
   <name>remove-threat-windows</name>
   <executable>remove-threat.exe</executable>
   <timeout_allowed>no</timeout_allowed>
</command>

<active-response>
   <disabled>no</disabled>
   <command>remove-threat-windows</command>
   <location>local</location>
   <rules_id>100003</rules_id>
</active-response>

Where:

• name specifies the name of the command being called in the active response section.
• executable specifies the executable file to run. In this case remove-threat.exe.
• The <active response> block calls the command block when the rule ID specified triggers an alert.

Testing the active response configuration

To test the active response configuration, we proceed to download the eicar.com test file on the agent where the remove-threat.exe binary is.

We can see that the file is deleted upon download.

When we check the active-responses.log file, we see the entry below showing that the threat was successfully removed:

2022/06/14 14:29:42 active-response/bin/remove-threat: {"command": "add", "version": 1, "parameters": {"program": "active-response/bin/remove-threat", "alert": {"manager": {"name": "ubuntu-focal"}, "decoder": {"name": "syscheck_new_entry"}, "syscheck": {"event": "added", "inode_after": 141367, "mode": "realtime", "gid_after": "0", "uid_after": "0", "path": "/home/vagrant/eicar.com.txt.5", "size_after": "68", "mtime_after": "2020-07-01T19:34:06", "gname_after": "root", "sha256_after": "275a021bbfb6489e54d471899f7db9d1663fc695ec2fe2a2c4538aabf651fd0f", "md5_after": "44d88612fea8a8f36de82e1278abb02f", "perm_after": "rw-r--r--", "sha1_after": "3395856ce81f2b7382dee72602f798b642f14140", "uname_after": "root"}, "agent": {"name": "vagrant-ubuntu-trusty-64", "ip": "10.0.2.15", "id": "007"}, "rule": {"groups": ["local", "malware"], "firedtimes": 15, "description": "A file - /home/vagrant/eicar.com.txt.5 - in the malware blacklist was added to the system.", "mail": false, "level": 5, "id": "100002"}, "location": "syscheck", "timestamp": "2022-06-14T14:29:41.822+0000", "id": "1655216981.554271", "full_log": "File '/home/vagrant/eicar.com.txt.5' added\nMode: realtime\n"}, "extra_args": []}, "origin": {"module": "wazuh-execd", "name": "node01"}} Successfully removed threat

Creating rules for the active response log

We can create rules to alert when the active response file removal succeeded or failed. This is done by adding the following rule to the /var/ossec/etc/rules/local_rules.xml file on the manager then restarting it

<rule id="100004" level="7">
    <if_sid>657</if_sid>
    <match>Successfully removed threat</match>
    <description>$(parameters.program): Successfully removed threat $(parameters.alert.syscheck.path) whose MD5 hash appears in a malware blacklist.</description>
  </rule>
  <rule id="100005" level="7">
    <if_sid>657</if_sid>
    <match>Error removing threat</match>
    <description>$(parameters.program): Error removing threat $(parameters.alert.syscheck.path) whose MD5 hash appears in a malware blacklist.</description>
</rule>

This rule will create an alert with the result of the active response action i.e. “Successfully removed threat” or “Error removing threat” and the path to the file that was removed.

Testing the configuration

To test the configuration, we proceed to download the eicar test file on the monitored endpoint. We can see in the screenshot below that the file was deleted and we got an alert with the details of the active response.

Conclusion

In this post, we successfully used a CDB list to create a malware blacklist, then configured an active response to remove files whose MD5 hash are in the malware blacklist. CDB lists and active response can also be useful for configuring file control, for example, removing files from endpoints that are not in an allow list.