Detecting illegitimate crypto miners on Linux endpoints

Crypto miners are programs that utilize computer resources to mine cryptocurrency. Mining is the process that several cryptocurrencies use to generate new coins and verify new transactions. Crypto miners usually get rewarded a token for every successful transaction mined, which makes crypto mining a profitable activity.

The monetary gain of mining cryptocurrency is a motivation for threat actors to compromise the endpoints of unsuspecting users, and use them for illegitimate mining activities. The endpoints are usually part of a botnet that is being controlled by a Command and Control (CnC) server. 

The mode of initial access for threat actors may be through the compromise of SSH credentials or the exploitation of a vulnerability. Once the threat actors gain remote access to the endpoints, the crypto mining software is executed on it.

Some common Linux-based crypto mining botnets are PyCryptoMiner, Panchan, Lemon Duck, Sysrv, and HolesWarm.

Detecting crypto miners activity

In order to identify the indicators of compromise of the various crypto mining botnets, we analyzed commonly available crypto mining botnets and identified the following:

• The mode of initial access.
• The methods of delivering the crypto mining payload.
• The techniques used to gain persistence.
• The mode of scanning and infecting other endpoints on the network.

The typical flow of most crypto mining botnets is highlighted below:

1. Gain initial access to an endpoint via SSH or RCE.
2. Maintain persistence by:
   • Modifying the SSH keys file to add the threat actor’s key.
   • Creating cron jobs to schedule the crypto miner to run periodically.
3. Download and execute the crypto miner program on the endpoint. This is executed in the memory in many cases.
4. Scan other devices on the internet for weak credentials, or vulnerability for potential exploitation. The cycle repeats whenever a new endpoint is compromised.

The common artifacts indicative of potential crypto mining botnet activity are highlighted below:  

• Attempted SSH brute force for initial access.
• Multiple outbound SSH authentication failures. This can indicate when an endpoint has been compromised and is being used to launch attacks against others.
• Modification to SSH authorized_keys file to add the threat actor’s key. This is done to ensure that access is maintained on the endpoint even after the user changes the SSH password or authentication keys.
• Creating a cron job to maintain persistence.
• Disabling the firewall, or modification of firewall rules to allow connection to known crypto miner ports, and IP addresses.
• Unusual and continued high CPU usage on the endpoint. Crypto miners can be CPU intensive, therefore, high CPU usage can be indicative of the presence of one.
• Connection to known crypto mining pools. For example, MineXMR mining pool, Antpool, Nanopool, and F2Pool.
• Known crypto mining protocols are also indicative of crypto mining activity. The protocols used are not peculiar to crypto miners alone, they are used by several other applications. Therefore, deep packet inspection is necessary in order to detect crypto mining activity on the network more accurately.

1. Detecting initial access

To detect initial access, Wazuh already has rules to detect multiple failed SSH authentication attempts. It is important that the analysts pay close attention to endpoints with recurring authentication failures. An example of the alert raised by Wazuh during SSH authentication brute force is shown below:

The Wazuh dashboard also contains charts and statistics that immediately show the total number of failed authentication attempts over a period of time. This is useful in helping security analysts quickly identify anomalies.

A security recommendation to prevent password brute force is to use SSH keys rather than passwords.

Crypto miner botnets typically steal SSH keys from compromised endpoints and use them to authenticate to other endpoints on the network. In this case, the crypto mining botnet does not need to bruteforce the credentials, therefore it is important to ensure that all endpoints on the network are secure. The next line of defense in case there is a compromise is to monitor for other indicators of compromise such as the modification of SSH keys file amongst others.

The other common mode of initial access is via an exploited vulnerability. First of all, it is important to keep all system components up to date. The Wazuh Vulnerability Detector module can be used to detect vulnerabilities affecting the monitored endpoints. The identified vulnerabilities should be patched. In addition to detecting the vulnerabilities, it is also important to have rules that detect exploitation attempts of these vulnerabilities. This makes it possible to detect the early stage of the attack when the threat actors are trying to gain access to the endpoints being monitored.

2. Detect modification to SSH keys file

After gaining initial access, threat actors could add their keys to the SSH authorized_keys file in order to maintain persistence. To detect this activity, we use Wazuh file integrity monitoring (FIM) to monitor the authorized_keys file for changes. Changes to this file, especially on critical endpoints should be investigated further to ensure that they are done by legitimate users. To configure the FIM module to monitor the keys file, we do the following:

On the Wazuh server, edit the shared default agent configuration file at /var/ossec/etc/shared/default/agent.conf and add the following: 

<agent_config os="linux">
  <syscheck>
    <directories check_all="yes" realtime="yes">/home/*/.ssh/</directories>
    <directories check_all="yes" realtime="yes">/var/lib/*/.ssh/</directories>
    <directories check_all="yes" realtime="yes">/root/.ssh/</directories>
  </syscheck>
</agent_config>

We do this so that the configuration gets shared with a group of agents, which are the ones that will run the checks. In our case, we are sharing the policy with the default group, hence the agent.conf file is the one in the default directory.

In order to trigger an alert whenever there is a change to the keys file, we add the following rules to the /var/ossec/etc/rules/local_rules.xml file on the Wazuh server:

<group name="cryptominer,">

  <rule id="100010" level="10">
    <if_sid>554</if_sid>
    <field name="file"  type="pcre2">\/authorized_keys$</field>
    <regex type="pcre2">added</regex>
    <description>SSH authorized_keys file "$(file)" has been added.</description>
    <mitre>
      <id>T1098.004</id>
    </mitre>
  </rule>
  
  <rule id="100011" level="10">
    <if_sid>550</if_sid>
    <field name="file"  type="pcre2">\/authorized_keys$</field>
    <regex type="pcre2">modified</regex>
    <description>SSH authorized_keys file "$(file)" has been modified.</description>
    <mitre>
      <id>T1098.004</id>
    </mitre>
  </rule>

</group>

Restart the Wazuh manager to apply the new rules:

systemctl restart wazuh-manager

The following alerts are generated on the Wazuh dashboard when these rules are triggered:

While modification to the SSH keys file is not malicious in all cases, it is important to continuously monitor them for changes due to the security implications of a compromise.

3. Detect creation and modification of cron jobs

Wazuh by default has a set of rules to detect when changes are made to cron jobs. The rules are rules ID 2830, 2831, 2832, 2833, and 2834. The contents of the rules are shown below:

<!-- Cron messages -->
<group name="syslog,cron,">
  <rule id="2830" level="0">
    <program_name>crond|crontab</program_name>
    <description>Crontab rule group.</description>
  </rule>

  <rule id="2831" level="0">
    <if_sid>2830</if_sid>
    <match>^unable to exec</match>
    <description>Wrong crond configuration.</description>
  </rule>

  <rule id="2834" level="5">
    <if_sid>2830</if_sid>
    <match>BEGIN EDIT</match>
    <description>Crontab opened for editing.</description>
  </rule>

  <rule id="2832" level="5">
    <if_sid>2830</if_sid>
    <match>REPLACE</match>
    <description>Crontab entry changed.</description>
    <group>pci_dss_10.2.7,pci_dss_10.6.1,gpg13_4.13,gdpr_IV_35.7.d,hipaa_164.312.b,nist_800_53_AU.14,nist_800_53_AU.6,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,</group>
  </rule>

  <rule id="2833" level="8">
    <if_sid>2832</if_sid>
    <match>REPLACE (root)</match>
    <description>Root's crontab entry changed.</description>
    <mitre>
      <id>T1053.003</id>
    </mitre>
    <group>pci_dss_10.2.7,pci_dss_10.6.1,pci_dss_10.2.2,gpg13_4.13,gdpr_IV_35.7.d,gdpr_IV_32.2,hipaa_164.312.b,nist_800_53_AU.14,nist_800_53_AU.6,nist_800_53_AC.6,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,</group>
  </rule>

</group> <!-- SYSLOG,CRON -->

The alerts generated from these rules are shown below:

In order to investigate the alerts and understand exactly what modification has been made to the cron files, the analyst has to manually connect to the affected endpoint. We can make the investigative process more efficient by using the Wazuh FIM module to monitor the cron files with the report_changes option in the configuration. This option presents the textual content of the file that was modified on the Wazuh dashboard. This means that the analyst can view the exact changes that were made to the cron job from the Wazuh dashboard.

Cron job files are usually located in the /var/spool/cron/crontabs directory. We configure Wazuh to monitor this directory by adding the following configuration to the shared agent configuration file at /var/ossec/etc/shared/default/agent.conf on the Wazuh server:

<agent_config os="linux">
  <syscheck>
    <directories check_all="yes" realtime="yes" report_changes="yes">/var/spool/cron/crontabs/</directories>
  </syscheck>
</agent_config>

Next, we create a rule to alert and display the modification that has been made to the cron job file. The following rule is added to the cryptominer group in /var/ossec/etc/rules/local_rules.xml on the Wazuh server:

<rule id="100012" level="12">
  <if_sid>550, 554</if_sid>
  <field name="file" type="pcre2">^\/var\/spool\/cron\/crontabs</field>
  <description>Cron job has been modified for user "$(uname)". The following modification was made: "$(changed_content)"</description>
  <mitre>
    <id>T1053.003</id>
  </mitre>
</rule>

Restart the Wazuh manager to apply the new rule:

systemctl restart wazuh-manager

An alert generated from this rule is shown below:

The job that was scheduled can be seen from the alert description. We confirm from the Wazuh dashboard that the job is * * * * * /var/tmp/.ICE-unix/-l/sh >/dev/null 2>&1.

4. Detect dropped malware files

To detect known malware files, we can use the VirusTotal integration to scan new files that are added to the monitored endpoints. The FIM module is used in combination with the VirusTotal integration to detect malicious files. The FIM module calculates the hash of newly added files and prompts the VirusTotal integration to perform a lookup in the malware database.

The crypto mining payload is usually downloaded to the tmp/ directory,  and the Downloads/ directory of the compromised user account. To configure the FIM module to monitor these directories, we add the following configuration to the shared default agent configuration file at /var/ossec/etc/shared/default/agent.conf on the Wazuh server:

<agent_config os="linux">
  <syscheck>
    <directories check_all="yes" realtime="yes">/tmp/</directories>
    <directories check_all="yes" realtime="yes">/var/tmp/</directories>
    <directories check_all="yes" realtime="yes">/home/*/Downloads/</directories>
  </syscheck>
</agent_config>

To perform a lookup of the file hashes on VirusTotal, we add the following configuration to /var/ossec/etc/ossec.conf on the Wazuh server:

<integration>
  <name>virustotal</name>
  <api_key>API_KEY</api_key> <!-- Replace with your VirusTotal API key -->
  <group>syscheck</group>
  <alert_format>json</alert_format>
</integration>

The following alert appears on the Wazuh dashboard when a known malicious file is added to any of the monitored directories.

5. Monitoring CPU usage

Crypto miners are CPU intensive, therefore high CPU usage is indicative of a potential crypto miner running on an endpoint. We use the Wazuh command monitoring capability to periodically check the CPU usage of the monitored endpoints to detect the anomaly when the usage is high. To monitor CPU usage, we add the following configuration to the shared agent configuration file at /var/ossec/etc/shared/default/agent.conf on the Wazuh server:

<agent_config os="linux">
  <localfile>
    <log_format>full_command</log_format>
    <command>top -bn1 | grep "Cpu(s)" | sed "s/.*, *\([0-9.]*\)%* id.*/ class="EnlighterJSRAW" data-enlighter-language="xml" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group=""/" | awk '{print 100 -  class="EnlighterJSRAW" data-enlighter-language="xml" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="""%"}'</command>
    <alias>top cpu usage</alias>
    <frequency>60</frequency>
  </localfile>
</agent_config>

When setting up the log collector to execute remote commands in the shared agent configuration, you must enable remote commands for agent modules. Remote commands are disabled by default for security reasons so that the Wazuh server gains explicit consent from the agents before executing commands. 

Remote commands can be enabled by adding the following line to the /var/ossec/etc/local_internal_options.conf file on the monitored endpoint:

logcollector.remote_commands=1

Restart the Wazuh agent to apply this change:

systemctl restart wazuh-agent

Next, we create the rule to trigger an alert when the CPU usage is higher than 90%. The following rule is added to the cryptominer group in the local rule file located at /var/ossec/etc/rules/local_rules.xml on the Wazuh server:

<rule id="100013" level="12" ignore="600">
  <if_sid>530</if_sid>
  <match>ossec: output: 'top cpu usage</match>
  <regex type="pcre2">9\d%|100%</regex>
  <description>CPU usage higher than 90%.</description>
  <mitre>
    <id>T1496</id>
  </mitre>
</rule>

Restart the Wazuh manager to apply the new rule:

systemctl restart wazuh-manager

An alert triggered by this rule is shown below:

6. Detecting crypto mining network activity

Common crypto mining pools are: EasyHash, HashVault, MineXMR, XMRPool, and Nanopool. There are several approaches on Wazuh to monitor endpoints in order to detect connections to crypto mining pools. The most effective method of detecting this is by using a network intrusion detection system (NIDS) such as Suricata for its deep packet inspection capabilities.

The NIDS can be integrated with Wazuh and the alerts generated shown on the Wazuh dashboard. A central NIDS device can be configured to capture traffic for the entire network. The NIDS logs are then forwarded to the Wazuh server. To install Suricata on an Ubuntu 22.04 endpoint, we take the following steps:

1. Install Suricata and its dependencies on the Linux endpoint:

add-apt-repository ppa:oisf/suricata-5.0
apt-get update
apt install suricata

2. Download the Emerging Threats Open ruleset and replace the existing ruleset with it:

wget https://rules.emergingthreats.net/open/suricata-5.0.9/emerging.rules.tar.gz
tar zxvf emerging.rules.tar.gz
rm /etc/suricata/rules/* -f
mv rules/*.rules /etc/suricata/rules/ -f

3. Download and add the ruleset for detecting crypto miner activity:

 wget -O /etc/suricata/rules/crypto-Miners_public_pools.rules https://raw.githubusercontent.com/al0ne/suricata-rules/master/Crypto_miner_pool/crypto-Miners_public_pools.rules

4. Modify Suricata settings in the /etc/suricata/suricata.yaml file:

EXTERNAL_NET: "any"

default-rule-path: /etc/suricata/rules
rule-files:
- "*.rules"

Also, look for all mentions of eth0 in the settings file and replace it with the name of the network interface you wish to monitor:

af-packet:
  - interface: eth0
…
pcap:
  - interface: eth0
…
pfring:
  - interface: eth0

5. Enable and restart the Suricata service:

systemctl daemon-reload
systemctl enable suricata
systemctl restart suricata

6. Confirm that the Suricata service is running:

systemctl status suricata

7. Install and enroll the Wazuh agent on the endpoint if not previously installed.

8. Configure the Wazuh agent to forward Suricata events to the Wazuh server. This can be done by adding the following configuration to the agent configuration file at /var/ossec/etc/ossec.conf on the monitored endpoint:

<localfile>
  <log_format>json</log_format>
  <location>/var/log/suricata/eve.json</location>
</localfile>

9. Restart the Wazuh agent to apply the changes:

systemctl restart wazuh-agent

An alert generated from crypto miner network activity appears on the Wazuh dashboard as shown below:

Additionally, outbound SSH port scans in an attempt to identify and compromise other endpoints on the network can be detected with the aid of the Suricata integration.

Restart the Wazuh manager to apply all the configuration changes that have been made:

systemctl restart wazuh-manager

Conclusion

In summary, we have identified the tactics used by crypto mining botnets that target Linux endpoints. We highlighted the modes of initial access, the techniques used for persistence, and the mode of scanning and compromising other devices on the network. This post has highlighted the various Wazuh capabilities that can be used to detect the various stages of a compromise.

References

• Detection of cryptominers and mining botnets
• THREAT ALERT: Crypto miner attack – Sysrv-Hello Botnet targeting WordPress pods – Sysdig
• al0ne/suricata-rules: Suricata IDS rules