Detecting common Linux persistence techniques with Wazuh

February 23rd 2023 | by Henadence Anyam | Wazuh 4.3

Persistence techniques are mechanisms or configurations threat actors use to maintain illicit access to compromised endpoints after gaining initial access. Persistence guarantees that attackers have endpoint access regardless of system restarts, changed credentials, or other interruptions that may potentially terminate illegal access.

Once persistence techniques are set up, threat actors no longer have to perform exploitation steps to regain access to the endpoint. Examples of persistence techniques include the use of startup scripts, the creation of user accounts, and the modification of existing binaries.

This blog post covers a few common Linux persistence techniques an adversary uses to establish permanent access to compromised endpoints.

MITRE ATT&CK persistence techniques and detection

The MITRE Adversarial Tactics, Techniques, and Common Knowledge (ATT&CK) Framework provide real-world adversary tactics and techniques for modeling the behavior of threat actors. Attackers commonly use some of the following Linux persistence techniques:

T1098.004 – Account Manipulation: SSH Authorized Keys
T1037.004 – Boot or Logon Initialization Scripts: RC Scripts
T1136.001 – Create Account: Local Account
T1078.003 – Valid Accounts: Local Accounts
T1546.004 – Event Triggered Execution: Unix Shell Configuration Modification
T1574.006 – Hijack Execution Flow: Dynamic Linker Hijacking
T1053.003 – Scheduled Task/Job: Systemd Timers
T1505.003 – Server Software Component: Web Shell

Note

We implemented the detection techniques using Wazuh 4.3.10 and simulated the attacks against an Ubuntu 22.04 LTS endpoint.

T1098.004 – Account Manipulation: SSH Authorized Keys

This persistence technique uses SSH key-based authentication to maintain access to compromised endpoints. SSH, or secure shell, is an encrypted protocol used to administer and securely communicate with servers. SSH uses a file called authorized_keys that defines the SSH keys used for logging into a user account. You must configure this file in any user account that needs SSH keys for permanent access. The default configuration in most SSH implementations allows you to deploy new authorized keys.

Threat actors can leverage this technique to import their public keys into the file ~/.ssh/authorized_keys to establish communication with compromised endpoints at any point in time without specifying a password. Before threat actors can leverage this technique, they need to ensure PubkeyAuthentication is enabled in the SSH configuration file /etc/ssh/sshd_config. Additionally, threat actors need to ensure PermitRootLogin is enabled when logging in as root.

To detect this persistence technique with Wazuh, configure the file integrity monitoring (FIM) module to monitor the creation and modification of the ~/.ssh/authorized_keys and /etc/ssh/sshd_config files. The Wazuh FIM module monitors selected file paths on a filesystem and triggers alerts when files are created, modified, or deleted.

Perform the following steps on the Linux and Wazuh server endpoints to detect this persistence technique.

Note

Ensure to restart the Wazuh agent after creating a new user’s /.ssh/ directory so that the FIM module can adequately monitor it.

Linux endpoint configuration

1. Append the following configuration to the Wazuh agent /var/ossec/etc/ossec.conf file:

<ossec_config>
  <syscheck>
    <directories check_all="yes" realtime="yes">/root/.ssh/</directories>
    <directories check_all="yes" realtime="yes">/home/*/.ssh/</directories>
    <directories check_all="yes" realtime="yes">/var/*/.ssh/</directories>
    <directories check_all="yes" realtime="yes">/etc/ssh/sshd_config</directories>
  </syscheck>
</ossec_config>

2. Restart the Wazuh agent to apply the configuration changes:

$ sudo systemctl restart wazuh-agent

Wazuh server configuration

1. Create a custom rules file comm_persist_tech_rules.xml in the /var/ossec/etc/rules/ directory and add the following rules to trigger alerts for the SSH authorized keys persistence technique:

<group name="common_persistence_techniques,sshd,">
  <rule id="100100" level="10">
    <if_sid>554</if_sid>
    <field name="file" type="pcre2">\/authorized_keys$</field>
    <description>SSH authorized_keys file "$(file)" has been added</description>
    <mitre>
      <id>T1098.004</id>
    </mitre>
  </rule>
  
  <rule id="100101" level="10">
    <if_sid>550</if_sid>
    <field name="file" type="pcre2">\/authorized_keys$</field>
    <description>SSH authorized_keys file "$(file)" has been modified</description>
    <mitre>
      <id>T1098.004</id>
    </mitre>
  </rule>

  <rule id="100102" level="10">
    <if_sid>550</if_sid>
    <field name="file" type="pcre2">\/sshd_config$</field>
    <description>SSH config file "$(file)" has been modified</description>
    <mitre>
      <id>T1098.004</id>
    </mitre>
  </rule>
</group>

Where:

Rule ID 100100 is triggered when the ~/.ssh/authorized_keys file is added to the monitored endpoint.
Rule ID 100101 is triggered when the ~/.ssh/authorized_keys file is modified on the monitored endpoint.
Rule ID 100102 is triggered when the /etc/sshd_config file is modified on the monitored endpoint.

2. Restart the Wazuh manager to apply the configuration changes:

The following alerts are generated on the Wazuh dashboard when FIM detects changes in the monitored files.

T1037.004 – Boot or Logon Initialization Scripts: RC Scripts

Boot or Logon Initialization Scripts are scripts automatically executed at boot or logon initialization without user interaction. RC scripts include rc.local, rc.common, and other RC scripts specific to the Unix-like distribution.

The /etc/rc.local is a script that Linux administrators use to define startup scripts or custom services. However, the RC script has been replaced with systemd, and it is now mainly used for compatibility purposes. The generator systemd-rc-local-generator included in the current version of systemd checks for the existence of /etc/rc.local and executes it on boot. You should monitor this script, as modifications may be malicious.

Threat actors can leverage this script to create backdoors that give them continued access to compromised endpoints.

We use the following methods to detect this persistence technique:

Auditd: An auditing framework that collects and stores system events such as operating system calls, functions, and file access events. It is used to monitor the creation, modification, or deletion of the /etc/rc.local file.
Command monitoring: This Wazuh module monitors events that are not logged on the endpoint by executing commands and analyzing their output. It is used to monitor for suspicious process creation, for example, the /etc/rc.local process with a parent process ID (PPID) of 1. Processes with PPID of 1 indicate that the parent process terminated without waiting for the child to finish. It is also used to check the status of rc-local.service. A state other than the inactive state may indicate an attack attempt.

Perform the following steps on the Linux and Wazuh server endpoints to detect this persistence technique.

Linux endpoint configuration

1. Install “auditd” using the following command:

# apt install auditd -y

2. Add the following auditd rules to the /etc/audit/rules.d/audit.rules file to detect the creation, modification, or deletion of the /etc/rc.local file using the following command:

# echo "-w /etc/rc.local -p wa -k possible_rcscript_attack" >> /etc/audit/rules.d/audit.rules

3. Reload the auditd rules file and verify the above configuration is applied:

# auditctl -R /etc/audit/rules.d/audit.rules
# auditctl -l

4. Append the following configuration to the Wazuh agent /var/ossec/etc/ossec.conf file. This configuration forwards auditd logs to the Wazuh server and configures command monitoring.

<ossec_config>
  <!-- Forwarding auditd logs to the Wazuh server -->
  <localfile>
    <log_format>audit</log_format>
    <location>/var/log/audit/audit.log</location>
  </localfile>

  <!-- Command monitoring (command executes every 180 seconds) -->
  <localfile>
    <log_format>command</log_format>
    <command>ps -ef | grep "[/]etc/rc.local" | awk '{print $3}'</command>
    <alias>ppid of rc_local</alias>
    <frequency>180</frequency>
  </localfile>

  <localfile>
    <log_format>command</log_format>
    <command>systemctl show rc-local.service --property=ActiveState | awk -F"=" '{print $2}'</command>
    <alias>state of rc_local service</alias>
    <frequency>180</frequency>
  </localfile>
</ossec_config>

5. Restart the Wazuh agent to apply the configuration changes:

$ sudo systemctl restart wazuh-agent

Wazuh server configuration

1. Add the following rules to the /var/ossec/etc/rules/comm_persist_tech_rules.xml file to trigger alerts for this persistence technique:

<group name="common_persistence_techniques,">
  <rule id="100105" level="10">
    <if_sid>80700</if_sid>
    <field name="audit.key">possible_rcscript_attack</field>
    <description>[RC script "$(audit.file.name)" has been added, modified, or deleted]: Possible RC scripts persistence attack</description>
    <mitre>
      <id>T1037.004</id>
    </mitre>
  </rule>

  <rule id="100106" level="10">
    <if_sid>530</if_sid>
    <match>'ppid of rc_local': 1$</match>
    <description>[PPID of rc-local.service is 1]: Possible RC scripts persistence attack</description>
    <mitre>
      <id>T1037.004</id>
    </mitre>
  </rule>

  <rule id="100107" level="10">
    <if_sid>530</if_sid>
    <regex type="pcre2">'state of rc_local service': (?!inactive$)</regex>
    <description>[rc-local.service is not inactive]: Possible RC scripts persistence attack</description>
    <mitre>
      <id>T1037.004</id>
    </mitre>
  </rule>
</group>

Where:

Rule ID 100105 is triggered when the /etc/rc.local file is added, modified, or deleted on the monitored endpoint.
Rule ID 100106 is triggered when the PPID of rc-local.service is 1.
Rule ID 100107 is triggered when the status of rc-local.service is not inactive.

2. Restart the Wazuh manager to apply the configuration changes:

$ sudo systemctl restart wazuh-manager

The following alerts are generated on the Wazuh dashboard when the rules are triggered.

T1136.001 – Create Account: Local Account

This persistence technique involves the creation of new user accounts on compromised endpoints to maintain endpoint access. Adversaries can create users using the useradd, usermod, and passwd commands. Activities of these commands modify the /etc/shadow, /etc/gshadow, /etc/passwd, /etc/group, and /etc/login.defs files and are logged in /var/log/auth.log.

By default, Wazuh detects when a new user is added to Linux endpoints by monitoring user creation-related commands. However, threat actors can manually modify these files to create users without using the commands.

You can use the Wazuh FIM module to detect this technique by monitoring modifications to the above files.

Perform the following steps on the Ubuntu and Wazuh server endpoints to detect this persistence technique.

Linux endpoint configuration

1. Append the following configuration to the Wazuh agent /var/ossec/etc/ossec.conf file:

<ossec_config>
  <syscheck>
    <directories check_all="yes" realtime="yes">/etc/shadow</directories>
    <directories check_all="yes" realtime="yes">/etc/gshadow</directories>
    <directories check_all="yes" realtime="yes">/etc/passwd</directories>
    <directories check_all="yes" realtime="yes">/etc/group</directories>
    <directories check_all="yes" realtime="yes">/etc/login.defs</directories>
  </syscheck>
</ossec_config>

2. Restart the Wazuh agent to apply the configuration changes:

$ sudo systemctl restart wazuh-agent

Wazuh server configuration

1. Add the following rules to the /var/ossec/etc/rules/comm_persist_tech_rules.xml to trigger alerts for this persistence technique:

<group name="common_persistence_techniques,">
  <rule id="100115" level="10">
    <if_sid>550</if_sid>
    <field name="file" type="pcre2">\/etc\/passwd$|\/etc\/shadow$|\/etc\/gshadow$|\/etc\/group$|\/etc\/login.defs$</field>
    <description>[File "$(file)" has been modified]: Possible local account manipulation</description>
    <mitre>
      <id>T1136.001</id>
      <id>T1078.003</id>
    </mitre>
  </rule>
</group>

Where:

Rule ID 100115 is triggered when the /etc/shadow, /etc/gshadow, /etc/passwd, /etc/group, and /etc/login.defs files are modified.

2. Restart the Wazuh manager to apply the configuration changes:

$ sudo systemctl restart wazuh-manager

The following alerts are generated on the Wazuh dashboard when the rules are triggered.

T1078.003 – Valid Accounts: Local Accounts

Local accounts are those configured for users of an endpoint or service. They are locally stored on an endpoint and have assigned rights and permissions for the endpoint. Local accounts can be used for remote administration. This persistence technique involves the manipulation of local accounts to maintain persistence on compromised endpoints.

Cyber threat actors can modify existing account credentials to persist on compromised endpoints. Local accounts or users are stored in the /etc/passwd file of Linux endpoints. The usermod, chage, and passwd commands modify existing user account details, such as username, password, shells, and more. These commands modify the /etc/shadow, /etc/gshadow, /etc/passwd, /etc/group, and /etc/login.defs configuration files containing user account information.

You can monitor file modifications using the Wazuh FIM module to detect this persistence technique. By default, Wazuh detects remote SSH connections in case modified accounts are used to access compromised endpoints remotely.

The same rules in the T1136.001 – Create Account: Local Account provide coverage for this persistence technique.

T1546.004 – Event Triggered Execution: Unix Shell Configuration Modification

This persistence technique involves executing malicious commands through scripts triggered by a user’s shell. When Unix shells launch, they execute several configuration scripts. Threat actors can leverage these scripts to maintain persistence on compromised endpoints. They achieve this by inserting malicious commands that automatically get executed by shells. These scripts are /etc/profile, /etc/profile.d, /etc/bash.bashrc, /etc/bash.bash_logout, ~/.bash_profile, ~/.bash_login, ~/.profile, ~/.bash_profile, ~/.bashrc, and ~/.bash_logout.

To detect this persistence technique, monitor for changes in shell configuration scripts using the Wazuh FIM module.

Perform the following steps on the Linux and Wazuh server endpoints to detect this persistence technique.

Linux endpoint configuration

1. Append the following configuration to the Wazuh agent /var/ossec/etc/ossec.conf file:

<ossec_config>
  <syscheck>
    <directories check_all="yes" realtime="yes">/etc/</directories>
    <directories check_all="yes" realtime="yes">/home/*/.bash_profile</directories>
    <directories check_all="yes" realtime="yes">/home/*/.bash_login</directories>
    <directories check_all="yes" realtime="yes">/home/*/.profile</directories>
    <directories check_all="yes" realtime="yes">/home/*/.bash_profile</directories>
    <directories check_all="yes" realtime="yes">/home/*/.bashrc</directories>
    <directories check_all="yes" realtime="yes">/home/*/.bash_logout</directories>
    <directories check_all="yes" realtime="yes">/root/.bash_profile</directories>
    <directories check_all="yes" realtime="yes">/root/.bash_login</directories>
    <directories check_all="yes" realtime="yes">/root/.profile</directories>
    <directories check_all="yes" realtime="yes">/root/.bash_profile</directories>
    <directories check_all="yes" realtime="yes">/root/.bashrc</directories>
    <directories check_all="yes" realtime="yes">/root/.bash_logout</directories>
  </syscheck>
</ossec_config>

2. Restart the Wazuh agent to apply the configuration changes:

$ sudo systemctl restart wazuh-agent

Wazuh server configuration

1. Add the following rules to the /var/ossec/etc/rules/comm_persist_tech_rules.xml to trigger alerts for this persistence technique:

<group name="common_persistence_techniques,">
  <rule id="100120" level="10">
    <if_sid>554</if_sid>
    <field name="file" type="pcre2">\/etc\/profile$|\/etc/profile.d\/|\/etc\/bash.bashrc$|\/etc\/bash.bash_logout$|.bash_profile$|.bash_login$|.profile$|.bash_profile$|.bashrc$|.bash_logout$</field>
    <description>Unix shell config "$(file)" has been added</description>
    <mitre>
      <id>T1546.004</id>
    </mitre>
  </rule>
  
  <rule id="100121" level="10">
    <if_sid>550</if_sid>
    <field name="file" type="pcre2">\/etc\/profile$|\/etc/profile.d\/|\/etc\/bash.bashrc$|\/etc\/bash.bash_logout$|.bash_profile$|.bash_login$|.profile$|.bash_profile$|.bashrc$|.bash_logout$</field>
    <description>Unix shell config "$(file)" has been modified</description>
    <mitre>
      <id>T1546.004</id>
    </mitre>
  </rule>
</group>

Where:

Rule IDs 100120 and 100121 are triggered when the /etc/profile, /etc/profile.d, /etc/bash.bashrc, /etc/bash.bash_logout, ~/.bash_profile, ~/.bash_login, ~/.profile, ~/.bash_profile, ~/.bashrc, and ~/.bash_logout scripts are added or modified respectively.

2. Restart the Wazuh manager to apply the configuration changes:

$ sudo systemctl restart wazuh-manager

The following alert is generated on the Wazuh dashboard when the /root/.bashrc and /etc/bash.bashrc files are modified.

T1574.006 – Hijack Execution Flow: Dynamic Linker Hijacking

The dynamic linker is the program that loads shared objects (shared libraries) required by an executable. It loads the shared library into system memory and prepares the executable for use at runtime. The dynamic linker uses environment variables such as LD_PRELOAD, or configuration files such as /etc/ld.so.preload to load shared libraries.

Threat actors can trick the dynamic linker into loading malicious libraries by hijacking the execution flow of an application. The LD_PRELOAD environment variable and the /etc/ld.so.preload file usually contain user-specified shared objects to be loaded first by the dynamic linker. By default, the LD_PRELOAD environment variable is not set, and the /etc/ld.so.preload file does not exist in a vanilla installation of Linux. A malicious library is likely to be in use if this is the case.

To detect the Dynamic Linker Hijacking persistence technique, monitor the creation or modification of /etc/ld.so.preload using the Wazuh FIM module. You can also check if the LD_PRELOAD environment variable is set by using the command monitoring module.

Perform the following steps on the Linux and Wazuh server endpoints to detect this persistence technique.

Linux endpoint configuration

1. With auditd already installed and configured, add the following auditd rules to the /etc/audit/rules.d/audit.rules file to monitor the creation, modification, or deletion of the /etc/ld.so.preload file using the following command:

# echo "-w /etc/ld.so.preload -p wa -k possible_preload_hijack" >> /etc/audit/rules.d/audit.rules

2. Reload the auditd rules file and verify the above configuration is applied:

# auditctl -R /etc/audit/rules.d/audit.rules
# auditctl -l

3. Append the following configuration to the Wazuh agent /var/ossec/etc/ossec.conf file to monitor when the LD_PRELOAD environment variable is set:

<ossec_config>
  <!-- Command monitoring (command executes every 180 seconds) -->
  <localfile>
    <log_format>command</log_format>
    <command>printenv LD_PRELOAD</command>
    <alias>check for LD_PRELOAD envar</alias>
    <frequency>180</frequency>
  </localfile>
</ossec_config>

4. Restart the Wazuh agent to apply the configuration changes:

# systemctl restart wazuh-agent

Wazuh server configuration

1. Add the following rules to the /var/ossec/etc/rules/comm_persist_tech_rules.xml file to trigger alerts for this persistence technique:

<group name="common_persistence_techniques,">
  <rule id="100125" level="10">
    <if_sid>80700</if_sid>
    <field name="audit.key">possible_preload_hijack</field>
    <description>[Config file "ld.so.preload" has been added, modified, or deleted]: Possible dynamic linker hijacking</description>
    <mitre>
  	<id>T1574.006</id>
    </mitre>
  </rule>

  <rule id="100126" level="10">
    <if_sid>530</if_sid>
    <match>'check for LD_PRELOAD envar'</match>
    <description>[LD_PRELOAD envar has been set]: Possible dynamic linker hijacking</description>
    <mitre>
      <id>T1574.006</id>
    </mitre>
  </rule>
</group>

Where:

Rule ID 100125 is triggered when the /etc/ld.so.preload file is added, modified, or deleted from the monitored endpoint.
Rule ID 100126 is triggered when the LD_PRELOAD environment variable is set to preload a user-defined shared library.

2. Restart the Wazuh manager to apply the configuration changes:

$ sudo systemctl restart wazuh-manager

The following alerts are generated on the Wazuh dashboard when the rules are triggered.

T1053.003 – Scheduled Task/Job: Systemd Timers

Systemd timers are unit files used to schedule jobs in Linux environments. Timers can be used as an alternative to Cron. Systemd timers control system services and are appended with the file extension .timer. Systemd timers and systemd services work together in scheduling system services. This implies each .timer file must have a corresponding .service file with the same name. The files are generally stored in the /etc/systemd/system/, /usr/lib/systemd/system/, /usr/local/lib/systemd/system/, and /lib/systemd/system/ directories.

Threat actors can schedule the execution of malicious code by creating or modifying systemd timers.

To detect this persistence technique, configure the Wazuh FIM module to detect the creation or modification of .timer and .service files in the directories mentioned above.

Perform the following steps on the Linux and Wazuh server endpoints to detect this persistence technique.

Linux endpoint configuration

1. Append the following configuration to the Wazuh agent /var/ossec/etc/ossec.conf file:

<ossec_config>
  <syscheck>
    <directories check_all="yes" realtime="yes">/etc/systemd/system/</directories>
    <directories check_all="yes" realtime="yes">/usr/lib/systemd/system/</directories>
    <directories check_all="yes" realtime="yes">/usr/local/lib/systemd/system/</directories>
    <directories check_all="yes" realtime="yes">/lib/systemd/system/</directories>
  </syscheck>
</ossec_config>

2. Restart the Wazuh agent to apply the configuration changes:

$ sudo systemctl restart wazuh-agent

Wazuh server configuration

1. Add the following rules to the /var/ossec/etc/rules/comm_persist_tech_rules.xml file to trigger alerts for this persistence technique:

<group name="common_persistence_techniques,">
  <rule id="100130" level="12">
    <if_sid>554</if_sid>
    <field name="file" type="pcre2">\/systemd\/system\/.*\.timer$|\/systemd\/system\/.*\.service$</field>
    <description>[Systemd "$(file)" has been added]: Possible task/job scheduling</description>
    <mitre>
      <id>T1053.006</id>
    </mitre>
  </rule>
  
  <rule id="100131" level="12">
    <if_sid>550</if_sid>
    <field name="file" type="pcre2">\/systemd\/system\/.*\.timer$|\/systemd\/system\/.*\.service$</field>
    <description>[Systemd "$(file)" has been modified]: Possible task/job scheduling</description>
    <mitre>
      <id>T1053.006</id>
    </mitre>
  </rule>
</group>

Where:

Rule IDs 100130 and 100131 are triggered when .timer and .service files are added and modified respectively in the monitored /etc/systemd/system/, /usr/lib/systemd/system/, /usr/local/lib/systemd/system/, and /lib/systemd/system/ directories.

2. Restart the Wazuh manager to apply the configuration changes:

$ sudo systemctl restart wazuh-manager

The following alerts are generated on the Wazuh dashboard when the rules are triggered.

The detection for T1053.003 – Scheduled Task/Job: Cron can be found on section 3 of detecting illegitimate crypto miners on Linux endpoints.

T1505.003 – Server Software Component: Web Shell

A web shell is a malicious script that enables threat actors to compromise web servers and launch additional attacks. Threat actors use this technique to maintain persistent access to compromised endpoints.

Kindly visit our blog post on web shell attack detection with Wazuh for detailed information about its detection.

Conclusion

This blog post shows several techniques to establish persistence on compromised Linux endpoints. Following the MITRE attack framework, we covered how cyber threat actors use these Linux persistence techniques to maintain connectivity across compromised endpoints.

We have been able to detect the presence of persistence activities on Linux endpoints using the Wazuh FIM and command monitoring capabilities. Additionally, we integrated the Linux auditing system, auditd to monitor specific file events.

If you have any questions on this blog post or Wazuh in general, kindly join our Slack community.

Reference

1. MITRE ATT&CK persistence techniques.

Detecting common Linux persistence techniques with Wazuh

MITRE ATT&CK persistence techniques and detection

T1098.004 – Account Manipulation: SSH Authorized Keys

Linux endpoint configuration

Wazuh server configuration

T1037.004 – Boot or Logon Initialization Scripts: RC Scripts

Linux endpoint configuration

Wazuh server configuration

T1136.001 – Create Account: Local Account

Linux endpoint configuration

Wazuh server configuration

T1078.003 – Valid Accounts: Local Accounts

T1546.004 – Event Triggered Execution: Unix Shell Configuration Modification

Linux endpoint configuration

Wazuh server configuration

T1574.006 – Hijack Execution Flow: Dynamic Linker Hijacking

Linux endpoint configuration

Wazuh server configuration

T1053.003 – Scheduled Task/Job: Systemd Timers

Linux endpoint configuration

Wazuh server configuration

T1505.003 – Server Software Component: Web Shell

Conclusion

Reference

Detecting Living Off the Land attacks with Wazuh

Wazuh and Diamatix Forge Strategic Partnership to Bolster Cybersecurity Solutions

Kuiper ransomware detection and response with Wazuh

Wazuh and Wattle Tech sign a partnership agreement