Monitoring Windows resources with Performance Counters

| by Dario Menten | Wazuh 4.3
Post icon

Windows Performance Counters provide an in-depth and consistent interface for collecting different types of system data such as processor, memory, and disk usage statistics. Performance counters can be used to monitor system resources and performance.

This blog post describes how to use Wazuh to monitor Windows system resources like CPU, RAM, etc. Observing resource usage contributes to the overall security monitoring of endpoints. This is because anomalies in the performance of system resources can be an indicator of ongoing attacks.

Windows performance counter data can be viewed in real time with the perfmon utility.

Another way of viewing the performance counter data is through the Powershell cmdlet called Get-Counter. The Get-Counter cmdlet gets performance counter data directly from the performance monitoring instrumentation on Windows endpoints. We will focus this guide on the use of this cmdlet to obtain metrics from the endpoint.

List of available counters

Counters are used to inspect the usage of Windows system resources. With the use of the Get-Counter cmdlet in a Powershell console, we can list all available counters:

Get-Counter -ListSet *

This will list a lot of counters, and the information is usually truncated because of its size. To use this cmdlet, it is recommended to get a list of available counters for a counter set:

(Get-Counter -ListSet * | where {$_.CounterSetName -eq '<CounterSetName>'}).Paths

To list the counter sets, use the CounterSetName variable:

Get-Counter -ListSet * | Select <CounterSetName>

For example, if we need to query the available counters for the Memory counter set, we can run this command:

(Get-Counter -ListSet * | where {$_.CounterSetName -eq 'Memory'}).Paths

We will get a list of counter paths like this:

\Memory\Page Faults/sec
\Memory\Available Bytes
\Memory\Committed Bytes
\Memory\Commit Limit
\Memory\Write Copies/sec
\Memory\Transition Faults/sec
\Memory\Cache Faults/sec
\Memory\Demand Zero Faults/sec
\Memory\Pages/sec
\Memory\Pages Input/sec
\Memory\Page Reads/sec
\Memory\Pages Output/sec
\Memory\Pool Paged Bytes
\Memory\Pool Nonpaged Bytes
\Memory\Page Writes/sec
\Memory\Pool Paged Allocs
\Memory\Pool Nonpaged Allocs
\Memory\Free System Page Table Entries
\Memory\Cache Bytes
\Memory\Cache Bytes Peak
\Memory\Pool Paged Resident Bytes
\Memory\System Code Total Bytes
\Memory\System Code Resident Bytes
\Memory\System Driver Total Bytes
\Memory\System Driver Resident Bytes
\Memory\System Cache Resident Bytes
\Memory\% Committed Bytes In Use
\Memory\Available KBytes
\Memory\Available MBytes
\Memory\Transition Pages RePurposed/sec
\Memory\Free & Zero Page List Bytes
\Memory\Modified Page List Bytes
\Memory\Standby Cache Reserve Bytes
\Memory\Standby Cache Normal Priority Bytes
\Memory\Standby Cache Core Bytes
\Memory\Long-Term Average Standby Cache Lifetime (s)

These counter paths can be used to get the desired metrics.

Getting metrics with Powershell

Once we know which counter to query, we can run the cmdlet to obtain the information we want to measure. Each counter is uniquely identified through its name and its path (counter path), or location:

(Get-Counter '<CounterPath>').CounterSamples[0]

The result will be a Powershell object with some properties:

Path                                             InstanceName      CookedValue
----                                             ------------      -----------
\\win10-tools\processor(_total)\% processor time _total       3.14235688964537

With this information, we can configure a Wazuh command module to analyze performance metrics.

Configuring Wazuh to get counter metrics

The Wazuh command module allows the Wazuh manager to run remote commands and scripts on agents. Using this concept, we can write wodle commands to run Get-Counter cmdlet on agents and analyze the results with Wazuh.

The generic command for a counter:

	<wodle name="command">
		<disabled>no</disabled>
		<tag>CPUUsage</tag>
		<command>Powershell -c "@{ winCounter = (Get-Counter '<CounterPath>').CounterSamples[0] } | ConvertTo-Json -compress"</command>
		<interval>1m</interval>
		<ignore_output>no</ignore_output>
		<run_on_start>yes</run_on_start>
		<timeout>0</timeout>
	</wodle>

To get the <CounterPath>, you can refer to previous points in this section. The tags used in this wodle command are explained in this documentation. You can apply this configuration directly in the Wazuh agent configuration file (C:\Program Files (x86)\ossec-agent\ossec.conf) or you can distribute this configuration to a group of agents using the centralized configuration.

When using a centralized configuration, the wodle configuration is placed in the manager /var/ossec/etc/shared//agent.conf file. Setting up remote commands in the shared agent configuration must be enabled explicitly for each agent. Otherwise, that agent will ignore the command to run. To enable remote command execution, add the following line to the C:\Program Files (x86)\ossec-agent\local_internal_options.conf file on the agent:

wazuh_command.remote_commands=1

As an example, we can choose a counter path (\Processor()\% Processor Time) and then execute this command (@{ winCounter = (Get-Counter '\Processor()\% Processor Time').CounterSamples[0] } | ConvertTo-Json -compress) in Powershell on the Windows system. This will create a piece of log in JSON format that will be sent through the agent to the Wazuh manager. We can see an example:

{"winCounter":{"Path":"\\\\win10-tools\\processor(_total)\\% processor time","InstanceName":"_total","CookedValue":90.5432,"RawValue":302816679687,"SecondValue":132956595027922186,"MultipleCount":1,"CounterType":558957824,"Timestamp":"\/Date(1651185902792)\/","Timestamp100NSec":132956487027920000,"Status":0,"DefaultScale":0,"TimeBase":10000000}}

In order to generate a Wazuh alert with this log, we need to create rules to match it.

Generate the alert with a rule

Wazuh has default and custom rulesets that are used to analyze incoming events and generate alerts when appropriate. Refer to the rules syntax, and custom rules and decoders sections of Wazuh documentation to understand how rules are created in the Wazuh manager.

Parent rule to catch counters logs

This is a parent rule that will analyze logs created by the wodle command and generate alerts on the Wazuh dashboard. We recommend creating a new custom rules file based on this parent rule.

<group name="WinCounter,">

    <rule id="301000" level="3">
	  <decoded_as>json</decoded_as>
	  <match>^{"winCounter":</match>
	  <description>Windows Performance Counter: $(winCounter.Path)</description>
    </rule>
    
</group>

Use case: CPU and memory metrics

As an example of what we can achieve with this, we set up an environment to get metrics from counters related to memory and CPU usage. The environment is made up of a Wazuh manager and an enrolled agent installed on a Windows endpoint.

Step 1: On the manager, set up a group that will contain the Windows endpoints to be monitored. Place the configuration below in the /var/ossec/etc/shared//agent.conf file.

Agent configuration:

<agent_config>
	<!-- Shared agent configuration here -->
	<wodle name="command">
		<disabled>no</disabled>
		<tag>CPUUsage</tag>
		<command>Powershell -c "@{ winCounter = (Get-Counter '\Processor(_Total)\% Processor Time').CounterSamples[0] } | ConvertTo-Json -compress"</command>
		<interval>1m</interval>
		<ignore_output>no</ignore_output>
		<run_on_start>yes</run_on_start>
		<timeout>0</timeout>
	</wodle>
	<wodle name="command">
		<disabled>no</disabled>
		<tag>MEMUsage</tag>
		<command>Powershell -c "@{ winCounter = (Get-Counter '\Memory\Available MBytes').CounterSamples[0] } | ConvertTo-Json -compress"</command>
		<interval>1m</interval>
		<ignore_output>no</ignore_output>
		<run_on_start>yes</run_on_start>
		<timeout>0</timeout>
	</wodle>
</agent_config>

Step 2: Write custom rules to generate alerts about system performance. A new rule file called performance_monitor.xml was created:

Rules configuration:

<group name="WinCounter,">

    <rule id="301000" level="0">
	  <decoded_as>json</decoded_as>
	  <match>^{"winCounter":</match>
	  <description>Windows Performance Counter: $(winCounter.Path)</description>
    </rule>
    
    <rule id="302000" level="3">
	  <if_sid>301000</if_sid>
	  <field name="winCounter.Path">memory\\available mbytes</field>
	  <description>Windows Counter: Available Memory</description>
	  <group>MEMUsage,</group>
    </rule>

    <rule id="302001" level="5">
	  <if_sid>302000</if_sid>
	  <field name="winCounter.CookedValue" type="pcre2">^[5-9]\d\d$</field>
	  <description>Windows Counter: Available Memory less than 1GB</description>
	  <group>MEMUsage,</group>
    </rule>
    
    <rule id="302002" level="7">
	  <if_sid>302000</if_sid>
	  <field name="winCounter.CookedValue" type="pcre2">^[1-4]\d\d$</field>
	  <description>Windows Counter: Available Memory less than 500GB</description>
	  <group>MEMUsage,</group>
    </rule>
    
    <rule id="303000" level="3">
	  <if_sid>301000</if_sid>
	  <field name="winCounter.Path">processor\S+ processor time</field>
	  <description>Windows Counter: CPU Usage</description>
	  <group>CPUUsage,</group>
    </rule>

    <rule id="303001" level="5">
	  <if_sid>303000</if_sid>
	  <field name="winCounter.CookedValue">^8\d.\d+$</field>
	  <description>Windows Counter: CPU Usage above 80%</description>
	  <group>CPUUsage,</group>
    </rule>
    
    <rule id="303002" level="7">
	  <if_sid>303000</if_sid>
	  <field name="winCounter.CookedValue">^9\d.\d+$</field>
	  <description>Windows Counter CPU Usage above 90%</description>
	  <group>CPUUsage,</group>
    </rule>

</group>

Rules description:

  • Rule 301000 (level 0): This is the base rule to catch all the logs from the winCounter commands. This will serve as the base for other rules in the WinCounter group.
  • Rule 302000 (level 3): This is the parent rule for rules 302001 and 302002. It is used to monitor all the metrics from the counter path: memory\available mbytes (MEMUsage metrics).
  • Rule 302001 (level 5) and Rule 302002 (level 7): These rules generate alerts for certain thresholds in memory usage.
  • Rule 303000 (level 3): This is the parent rule for rules 303001 and 303002. It is used to monitor all the metrics from the counter Path: processor(_total)\% processor time (CPUUsage metrics).
  • Rule 303001 (level 5) and Rule 303002 (level 7): These rules generate alerts for certain thresholds in CPU usage.

After adding these rules and restarting the Wazuh manager service, alerts will start to be displayed on the Wazuh dashboard:

Building a dashboard

Taking into account the PoC above, you can do more than just alerting. Metrics can be displayed on custom dashboards and graphs to present a better picture of system performance.

Modifying the Wazuh Template

By default, the Wazuh indexer will analyze values from these alerts as string data types. In order to use the alerts to create visualizations and dashboards, we need to set them to the long data type.

Step 1: Adding the fields in the template.

Modify the following file: /etc/filebeat/wazuh-template.json and add these lines to the data section:

          "winCounter": {
             "properties": {
               "CookedValue": {
                 "type": "long"
               },
               "RawValue": { 
                 "type": "long"
               }
             }
          },

Step 2: Applying the template.

In order to apply the changes to the template we have in the Wazuh indexer, we need to run the following command:

filebeat setup -index-management

Once this is applied, we can build custom visualizations and dashboards with those values:

Conclusion

This guide details how Wazuh can be used to monitor Windows system resources like CPU, RAM, etc. Observing resource usage contributes to the overall security monitoring of endpoints.

References