How Wazuh provides endpoint security without kernel-level access
User mode and kernel mode are two operating states within a computer system that define different levels of access and control to the hardware resources of a computer. Choosing the right mode between the two is important, as it affects the security and stability of the computer.
User mode is a restricted operating environment where applications run with limited access to system resources such as memory, CPU, and I/O devices. In this mode, processes operate in a controlled space, preventing them from directly interacting with hardware or critical system components. This restriction ensures that errors or crashes within user mode processes do not compromise the system’s stability.
On the other hand, kernel mode is where the operating system’s core components, such as the kernel, device drivers, and hardware management routines, operate. Applications in this mode can access all system resources, including memory, hardware devices, and CPU instructions. Kernel mode is necessary for executing low-level tasks that require complete control over the system, such as managing memory, processing interrupts, and performing I/O operations.
Security monitoring programs can operate in either user or kernel modes depending on their design. User-mode programs have fewer privileges than those operating in kernel mode. Kernel mode programs can detect and handle advanced, low-level malware that user-mode security solutions cannot reach, like boot kits or stealthy system modifications. However, programs that run in the kernel mode have several significant drawbacks. These programs operate with elevated privileges, making any vulnerabilities or malicious code within them potentially catastrophic for the entire system.
Difference between kernel mode and user mode applications
Security monitoring with kernel mode provides deep visibility into the inner workings of an operating system. This level of monitoring enables the detection of threats and anomalies that might evade traditional security measures. However, it also introduces potential risks, in the table below, we compare kernel mode and user mode using various criteria.
| Criteria | Kernel mode | User mode |
| Privilege level | Highest privilege level. | Lower privilege level. |
| Access | Full access to hardware and memory. | Restricted access; needs system calls to interact with hardware. |
| Stability | Less stable, as errors can crash the system. | More stable, errors typically only affect the running application. |
| Security | Less secure, vulnerabilities can impact the entire system. | More secure, as vulnerabilities are usually isolated to individual processes. |
This article examines how Wazuh provides endpoint security while running in user mode. Security monitoring in user mode reduces complexity and avoids potential security risks associated with kernel-level access.
How Wazuh monitors endpoints with user-space access
Wazuh leverages several Application Programming Interfaces (APIs) and Operating System (OS) services to detect threats and provide security insights. Wazuh collects and analyzes logs, file changes, running processes, and network activity from monitored endpoints to identify and respond to anomalies and suspicious activities.
This section details the APIs and OS services used by the Wazuh agent and highlights its key capabilities that ensure effective protection of endpoints without needing kernel-level access.
APIs and OS services used by the Wazuh agent core
An application core refers to essential components containing the fundamental logic and functionality of the application. In this section, we discuss the APIs and OS services used by the Wazuh agent core under the following criteria:
Multithreading
Multithreading is a programming concept that divides a process into smaller, independent tasks called threads that can run concurrently. Each thread represents a separate execution path within a program, allowing multiple operations to happen simultaneously. Wazuh uses the POSIX Threads (PThreads) library for thread management. The POSIX Threads (PThreads) library creates and manages multiple threads within a single process in Unix-like operating systems. PThreads allow the Wazuh agent to execute concurrent tasks, improving performance and efficiency.
Agent communication
The Wazuh agent communicates with the Wazuh server to transmit collected data and security events. Additionally, the agent provides operational data, including its configuration and status. Once connected, the agent can be monitored, upgraded, and configured remotely from the Wazuh server. Wazuh uses the POSIX sockets library and Windows Sockets 2 API for agent communication.
- POSIX sockets library
The POSIX sockets library is used for network communication in Unix-like operating systems, including Linux and macOS. It provides an API for creating, configuring, and managing network connections, allowing programs to communicate over networks using standard protocols such as TCP/IP and UDP. This library enables the Wazuh agent on Unix-like endpoints to communicate with the Wazuh central components.
- Windows Sockets 2 API
Windows Sockets 2 is an API used for network communication in Windows operating systems. It provides an interface for applications to connect to the internet or communicate over a network using different network protocols, such as TCP/IP, UDP, and others. This API allows the Wazuh agent on Windows endpoints to communicate with the Wazuh central components.
APIs and OS services used by Wazuh capabilities
This section discusses the APIs and OS services used by the Wazuh SIEM and XDR capabilities to ensure security monitoring while running in user mode.
Log data collection
Log data collection is the process of gathering and centralizing log files from various sources within a network, such as servers, applications, firewalls, and endpoints. These logs record events and activities within the network, providing valuable information for security analysis.
Wazuh uses the Logcollector module to collect logs from monitored endpoints. These logs are then analyzed in real-time by the Wazuh server using decoders and rules. The APIs and OS services used by the Wazuh Logcollector module include:
- Windows Event Log API
The Windows Event Log API provided by Microsoft allows applications and system components to interact with the Windows Event Logging service. The service manages event logs from software and hardware. The Wazuh agent uses the Windows Event Log API to get logs from Windows event channels, which are then sent to the Wazuh server for analysis.
The default Wazuh configuration on Windows includes the setting to monitor the Application logs from the Windows event channels:
<localfile>
<location>Application</location>
<log_format>eventchannel</log_format>
</localfile>
You can view Application logs by navigating to the Windows Logs section in Windows Event Viewer.
You can view application logs of the monitored endpoint on the Wazuh dashboard using the filter data.win.system.channel: Application, as seen in the image below.
The rule description from the event with rule ID 60702 matches the VSS event from the Windows event viewer.
- Unified Logging System (ULS)
The macOS Unified Logging System (ULS) consolidates log management and storage across all system levels. ULS is available in macOS 10.12 and later. Since macOS ULS does not write data to text-based log files, Wazuh collects logs from macOS endpoints using the Command Line Interface (CLI) log tool. The logs are then sent to the Wazuh server for analysis.
The default Wazuh configuration on macOS includes the setting to monitor SSH, security sessions, and authentication events using ULS:
<localfile> <location>macos</location> <log_format>macos</log_format> <query type="trace,log,activity" level="info">(process == "sudo") or (process == "sessionlogoutd" and message contains "logout is complete.") or (process == "sshd") or (process == "tccd" and message contains "Update Access Record") or (message contains "SessionAgentNotificationCenter") or (process == "screensharingd" and message contains "Authentication") or (process == "securityd" and eventMessage contains "Session" and subsystem == "com.apple.securityd")</query> </localfile>
Run the command below on a monitored macOS endpoint to see only sudo events:
log stream --process="sudo"
When a user attempts to run sudo with an incorrect password 3 times, the CLI log tool shows this log event:
2024-11-01 12:10:32.088100+0100 0x1b0b25 Default 0x0 38316 0 sudo: wazuh : 3 incorrect password attempts ; TTY=ttys001 ; PWD=/Users/wazuh ; USER=root ; COMMAND=/usr/bin/su
These events trigger alerts with rule ID 5404 on the Wazuh dashboard, as seen in the image below.
Expanding the alert provides information about the sudo event and displays a log event that matches the CLI log tool’s output.
- Journald
Journald is a part of the systemd system and service manager used in Linux distributions. The journald service replaces the older syslog daemons and provides a centralized and structured approach to logging. It collects and manages log data from various sources, including system services, the kernel, and user applications. The Wazuh agent collects logs using the journald service and forwards them to the Wazuh server for analysis.
The default Wazuh configuration on Linux endpoints includes the setting to collect logs from the journald service:
<localfile>
<log_format>journald</log_format>
<location>journald</location>
</localfile>
You can check the journald service logs for failed authentication with the command below:
journalctl -u ssh | grep "Failed"
Our monitored endpoint shows the following output:
Nov 08 17:55:57 wazuh-fluent-virtual-machine sshd[10877]: Failed password for invalid user admin from 192.168.33.1 port 31487 ssh2 Nov 08 17:56:02 wazuh-fluent-virtual-machine sshd[10877]: Failed password for invalid user admin from 192.168.33.1 port 31487 ssh2 Nov 08 17:56:06 wazuh-fluent-virtual-machine sshd[10877]: Failed password for invalid user admin from 192.168.33.1 port 31487 ssh2 Nov 08 17:56:14 wazuh-fluent-virtual-machine sshd[11027]: Failed password for invalid user admin from 192.168.33.1 port 31513 ssh2 Nov 08 17:56:17 wazuh-fluent-virtual-machine sshd[11027]: Failed password for invalid user admin from 192.168.33.1 port 31513 ssh2
The image below shows an alert with rule.id 5710, which shows the failed SSH authentication attempt. You can also see the log source, which is the journald service.
Malware detection
Wazuh uses the Rootcheck module to detect anomalies that might indicate the presence of rootkits and malware in an endpoint. Rootkits are particularly dangerous as they can give unauthorized users root privileges, allowing them to hide their activities and take full control of a system. The Wazuh Rootcheck module uses the following APIs and OS services to detect malware and rootkits:
- Proc pseudo-filesystem
The proc (process) pseudo-filesystem (usually mounted at /proc) is a virtual filesystem in Unix-like operating systems that provides a mechanism to interact with and obtain information about running processes and the system’s kernel. The Wazuh Rootcheck module uses this virtual file system to detect malicious processes that conceal themselves from the system’s list of running processes.
- Netstat
Malware often communicates using unconventional or hidden ports. The Wazuh Rootcheck module can identify these by attempting to bind to all system ports. If it fails to bind to a port and netstat doesn’t report it as in use, it could indicate that malware is covertly operating on that port.
- Win32 API
The Win32 API is a core set of system-level functions provided by Windows for interacting with the operating system. It allows developers to manage processes, files, memory, and Windows. The Wazuh Rootcheck module uses several functions within this API, such as CreateToolhelp32Snapshot, Process32First, and Process32Next, to gather detailed information about running processes. Process enumeration helps identify suspicious or unknown processes that might be rootkits.
File integrity monitoring
The Wazuh File Integrity Monitoring (FIM) module monitors files and directories and triggers an alert when a user or process creates, modifies, and deletes monitored files.
The module uses the following API and OS services:
- inotify
The inotify Linux kernel subsystem acts as a file system event monitor. It monitors file system events such as creation, deletion, and modification. The API has several system calls used by Wazuh, such as inotify_init, inotify_add_watch, and inotify_rm_watch. These system calls are used to initialize a new inotify instance, add a watch to an inotify instance for a specified file or directory, and remove a watch from an inotify instance. The Wazuh FIM module uses these calls in inotify to detect changes to monitored files or directories in real-time on Linux endpoints.
- Auditd
Auditd is the Linux Audit Daemon. It is a userspace component of the Linux Auditing System that monitors and records system activities based on rules defined in the /etc/audit/audit.rules file. The Wazuh FIM module uses both inotify and auditd for who-data monitoring. This allows the module to record information about who or what processes made changes to the monitored files or directories on Linux endpoints.
In the example below, we monitor the /etc/ssh/sshd_config file with the who-data option. This ensures that when the SSH configuration is modified, an alert that includes audit information about who and what process made changes to the file is triggered:
<directories check_all="yes" report_changes="yes" whodata="yes">/etc/ssh/sshd_config</directories>
The following command checks if the audit rule for monitoring the selected directory is applied:
auditctl -l | grep wazuh_fim
The output shows that the Wazuh FIM audit rule was added:
-w /etc/ssh/sshd_config -p wa -k wazuh_fim
Where:
-w /etc/ssh/sshd_config: Sets a watch on the monitored/etc/ssh/sshd_configfile, so any access or modifications to it will trigger an audit log entry.-p wa: Defines the permissions being monitored.w (write): Logs events when the file is modified.a (attribute change): Logs events when file attributes (such as permissions) are changed.
-k wazuh_fim: Sets a custom keywazuh_fimfor easier filtering and searching in the audit logs.
When the SSH configuration is changed, you can visualize the alert data in the Wazuh dashboard. To do this, go to Endpoint security > File Integrity Monitoring module and add the filter syscheck.path: is /etc/ssh/sshd_config in the search bar to query the alerts.
You can expand the alert to see more information such as the user and the process that modified the file and the modified content. In this example, the wazuh user changed the SSH port from 22 to 2202 using nano text editor.
- Win32 API
The Win32 API comprises essential system-level functions offered by Windows for OS interaction. It allows developers to handle processes, files, memory, and Windows management. The Wazuh agent uses this API to manage the Access Control List (ACL) and Windows event logs. This API allows the Wazuh agent to monitor file changes using real-time and who-data monitoring on Windows endpoints.
- Windows Registry
The Windows Registry is a hierarchical database in Microsoft Windows that holds configuration settings, options, and information for the operating system, applications, and hardware devices. The Wazuh FIM module scans the Windows Registry periodically and triggers an alert when it detects changes in the monitored entries.
We use the configuration below to monitor and record changes to the startup registry keys. The Run and RunOnce registry keys in Windows are used to automatically execute programs during the startup process. They can be useful for legitimate applications but are also commonly exploited by malware to persist on a system:
<syscheck>
<windows_registry arch="both" report_changes="yes">HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run</windows_registry>
<windows_registry arch="both"report_changes="yes">HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnce</windows_registry>
</syscheck>
When a new registry key is added to the Run and RunOnce registry keys, the event triggers alerts with rule ID 752 on the Wazuh dashboard, as seen in the image below. You can also see the affected registry key and the changed value.
System inventory
A system inventory contains information about an IT infrastructure’s hardware and software assets. Maintaining system inventory enhances an organization’s visibility into its hardware and software landscape.
The Wazuh Syscollector module collects system information from each monitored endpoint. The image below shows data collected by the Wazuh agent from a monitored endpoint, including operating system details, hardware, processes, network interfaces, and packages.
This information is sent to the Wazuh server for analysis and can be used to identify vulnerable applications and malicious artifacts.
The Wazuh agent uses the following APIs and OS services to gather system inventory information:
- Windows Registry
The Windows Registry is a hierarchical database in Microsoft Windows that stores configuration settings, options, and information for the operating system, applications, and hardware devices. When you install a program, Windows creates a new subkey in the registry. This subkey contains information such as the program location, version, and startup instructions. The Wazuh agent scans the Windows Registry to gather inventory data, such as the installed updates sent to the Wazuh server for analysis.
- DPKG & RPM
Debian Package Manager (DPKG) and RPM Package Manager (RPM) are both package management systems used in different Linux distributions to handle the installation, updating, and removal of software packages. DPKG is used in Debian-based Linux distributions such as Ubuntu, while RPM is used in Red Hat Enterprise Linux (RHEL), CentOS, and Fedora. The Wazuh agent uses these package managers to gather system inventory information about installed programs, which is sent to the Wazuh server for analysis.
- IOKit
IOKit is a framework designed by Apple to develop device drivers on macOS and iOS. It facilitates device access by allowing applications and services to interact with hardware devices and drivers through a device-interface mechanism. The Wazuh agent uses this framework to gather information about connected hardware devices on a macOS endpoint.
- WMI
Windows Management Instrumentation (WMI) is a framework Microsoft provides for managing data and operations on Windows-based operating systems. It allows users to automate administrative tasks, such as querying system information, configuring settings, and monitoring system events.
- WUA
The Windows Update Agent (WUA) API is a powerful interface that allows applications to interact with the Windows Update service. It provides functionalities to check for, download, and install updates, ensuring that systems remain up-to-date with the latest security patches and features. Through the WUA API, applications can manage update settings, schedule updates, and retrieve detailed information about installed updates. The Wazuh agent uses this API to retrieve system inventory data about installed updates on a Windows endpoint.
Vulnerability detection
Vulnerability detection enables organizations to proactively identify and address weaknesses in their systems before attackers can exploit them. This approach reduces the attack surface, minimizing the likelihood of a successful cyber breach. Additionally, regular vulnerability assessments are often required for compliance with industry regulations, helping organizations manage risks and avoid potential penalties.
Wazuh detects vulnerabilities by collecting a list of installed applications using the system inventory module and periodically sending this information to the Wazuh server. The server stores this data in local SQLite databases. The Vulnerability Detection module on the Wazuh server then compares the software inventory with vulnerability content documents to identify any vulnerable software on the monitored endpoints.
In the image below, the Wazuh vulnerability detection shows that a vulnerable version of Django is installed on the endpoint.
When the vulnerability is remediated, Wazuh updates the vulnerability status as solved as seen in the image below.
The Wazuh vulnerability detection capability enables organizations to detect vulnerable software and operating systems within their infrastructure, allowing them to remediate these vulnerabilities before attackers exploit them.
Container monitoring
Containers provide a lightweight, portable, and consistent environment for applications to run. It allows developers to deploy applications more efficiently and respond quickly to changing demands.
Container monitoring helps maintain containerized applications’ health, performance, and security. Wazuh uses the Docker Engine API to provide visibility in a containerized environment. The Docker Engine API is a RESTful API that allows applications to interact with the Docker daemon. The API enables Wazuh to monitor changes to container images, network configurations, and data volumes. Additionally, Wazuh can generate alerts for containers running in privileged mode and track users executing commands within running containers using this API.
In the image below, Wazuh detects several Docker activities, such as starting an instance, running some Docker commands, and deleting the container.
Monitoring system calls
Monitoring system calls on Linux endpoints provides information for security auditing, as it helps security teams identify suspicious behavior patterns and investigate potential incidents promptly. The Linux Audit system is a powerful tool for collecting both security and non-security events on Linux endpoints. However, the vast amount of data generated by audit logs can overwhelm system administrators, making it challenging to identify potential security threats and violations.
Wazuh uses the Linux Audit system to monitor system calls on Linux endpoints. The Wazuh agent installs and configures audit rules on monitored endpoints to collect relevant system call events, which are then sent to the Wazuh server for analysis. These audit rules capture critical security events, and Wazuh provides out-of-the-box detection rules to identify activities such as file access, command execution, privilege escalation, and malware.
In the image below, we show how the Auditd integration allows monitoring of suspicious commands executed on a Linux endpoint.
Security Configuration Assessment
The Wazuh Security Configuration Assessment (SCA) capability helps ensure your systems comply with predefined security policies and configuration settings. Wazuh incorporates industry-standard checks such as the CIS benchmarks, allowing you to check if your endpoints conform to these best practices.
Wazuh conducts configuration assessments by checking an endpoint’s installed software, configuration, and system files to ensure compliance with best practices. These checks are called SCA policies and are written in YAML format, making it easy for users to read and write their own SCA policies. By using these non-intrusive methods, Wazuh effectively evaluates system security while minimizing the risk of impacting system stability or performance.
In the image below, we carry out a configuration assessment on a Ubuntu endpoint. You can see information such as rationale, remediation, and a description of the check with ID 28575.
The endpoint failed the check because Uncomplicated Firewall (UFW), a host-based firewall is not running on the endpoint. When the ufw service is enabled and running on the endpoint, the result of the scan changes to passed, as seen in the image below.
Wazuh SCA continuously assesses the configuration of monitored endpoints to identify vulnerabilities, misconfigurations, and deviations from security baselines that attackers could exploit. It also provides detailed remediation steps to resolve the identified issues.
Incident response
Automated incident response is essential in security monitoring because it provides rapid, consistent, and scalable threat mitigation. It reduces the potential for human error and frees up resources for more complex tasks.
The Wazuh Active Response module can automatically execute predefined actions when threats are detected, such as blocking IP addresses, stopping malicious processes, and removing malicious files. Wazuh has out-of-the-box scripts that use applications found on endpoints to carry out actions such as blocking malicious network access and deleting malicious files on monitored endpoints. It offers customizable responses that integrate with detection rules and support multiple platforms, enabling organizations to neutralize threats and strengthen their security posture.
The image below shows the Wazuh Active Response module blocking malicious hosts performing a brute-force attack against a monitored CentOS endpoint.
Conclusion
This blog post examines how Wazuh monitors and secures endpoints without requiring invasive kernel-level access by leveraging existing APIs, system tools, and logs. This approach enhances security by avoiding the risks associated with kernel-mode programs, which can introduce severe and difficult-to-mitigate vulnerabilities.
Wazuh is a free and open source SIEM and XDR solution. You can deploy and manage Wazuh on-premises, or on the Wazuh cloud. Check out our community for support and updates.
References