Dynatrace Inc.
10/25/2023 | Press release | Distributed by Public on 10/25/2023 09:10
Detect VMware Aria Operations for Logs exploitation with Dynatrace and DQL
Earlier this week, virtualization services provider VMware alerted customers to the existence of a proof-of-concept (PoC) exploit for a recently patched security flaw in Aria Operations for Logs.
VMware Aria Operations for logs (formerly known as vRealize Log Insight) is used across enterprises to collect logs and provide analytics. Tracked as CVE-2023-34051 (CVSS score: 8.1), this high-severity vulnerability relates to a case of authentication bypass that could lead to remote code execution. Basically, this means that an unauthenticated malicious actor could inject files into the operating system of an impacted appliance which can result in remote code execution.
James Horseman from http://Horizon3.ai and the Randori Attack Team have been credited for the discovery and reporting. They have made a PoC for the vulnerability available and published the relevant indicators of compromise (IoC).
This vulnerability is a so-called "patch bypass" for a set of critical flaws that were addressed by VMware earlier this year, also discovered by Horizon3.ai. In their report they presented how an attacker could use three different CVEs to achieve remote code execution. Since the patch only blocks access to Thrift services by IP and did not fix the other CVEs in VMSA-2023-0001, all an attacker needs to do is spoof an IP address and use the published attack again.
In this blog post, we show how to discover the original attacks towards Aria Operations for Logs using Dynatrace and DQL by finding the IoC-s from the log records.
How to detect exploitation with Dynatrace
Since we're talking about a patch bypass, we're looking at the original attack vector described in this blog post. To exploit this vulnerability to gain remote code execution (RCE), the following steps have to be taken by the malicious actor:
- Create a Thrift client and you are allowed unauthenticated access to the Log Insight Thrift server.
- Spoof the IP address of the known worker.
- Create a malicious tar file containing a directory traversal using a valid file.
- Using , we upload the malicious Pak file to .
- We cause the Pak file to be extracted using . This writes our file to where we want on the filesystem.
Technical deep dive about the attack can be found here.
Discovering the attack with Dynatrace
To discover this attack, we have to go through the log files, that Aria Operations for Logs is creating. The Log Insight server stores relevant logs in the file . This log file is used to track all run time information related to Log Insight. To be able to analyse these logs with Dynatrace and DQL, one would have to Ingest the logs to Dynatrace beforehand.
Forwarding logs away from the main system is beneficial in case of any kind of successful attack by the malicious actor: if an attacker decides to remove all evidence from the system (including log files), then a security investigator wouldn't have traces left to analyse. Sending logs to Dynatrace on runtime would safeguard the log files for later analysis in a remote and secure location.
Security-related log records of Aria Operations for Logs are structured in the following format:
[2023-10-25 11:28:29.709+0000] ["https-jsse-nio-443-exec-9"/10.153.234.136 DEBUG] [com.vmware.loginsight.web.actions.misc.LoginActionBean] [User login success: vIDM: SAM=myusername, Domain=vmware.com, [email protected]]
When a new remote PAK file is being downloaded as a tarball, the following log lines are being created:
[com.vmware.loginsight.daemon.commands.SystemCommands] [PAK download initiated by node f2449ed5-11ee-45fd-a0a0-6225a33a8ac6] [com.vmware.loginsight.daemon.commands.SystemCommands] [Deleting existing PAK file: /tmp/exploit.pak] [com.vmware.loginsight.daemon.commands.SystemCommands] [Downloading http://192.168.4.133:8080/exploit.tar to /tmp/exploit.pak] [com.vmware.loginstght.daemon.commands.SystemCommands] [Max allowed pack size is 1505916450 bytes] [com.vmware.loginsight.daemon.commands.SystemCommands] [Current downloaded size is 21910] [com.vmware.loginsight.daemon.commands.SystemCommands] [Total downloaded size is 21910]
From these log lines, the best candidate to look for in the logs would be the third populated log. Lets fire up the DPL Architect and create a suitable pattern for this line:
With the created DPL pattern, we can extract the timestamp field , logger class name as that populated the log lines and the that was written down to the log record. These extracted fields would give us the possibility to simply filter out the relevant records:
But we had another goal in mind: we are after all the log records that contain the Downloading statement with the URL, from where the tarball was loaded from and the destination path in. As the blog post stated:
The URL, and filename will likely be different, however the filename will always have the format /tmp/<filename>.pak. Also note that this log may be legitimate. To determine if an attack has occurred, an administrator should evaluate the URL to determine if it is a legitimate download URL.
Let's continue with our DPL pattern and extract the full URL, where the payload was downloaded and the relevant filenames. Again, we're opening up the DPL Architect and create the following pattern:
'Downloading ' (ld («('/') [!/.]*:download_file '.tar')):url ' to /tmp/' (ld:filename '.pak'):saved_file
With the created pattern, we are extracting the and the tarballs' name as into separate fields using the look-around functionality of DPL. Additionally we're extracting the from the destination file path as well as the whole value for simpler analysis.
DPL pattern extracts the following result set from the log record:
Applying this pattern and filtering the records as instructed in the blogpost will give us the following result, that we can now pass on to the administrators for manual evaluation.
The longer description about how the exploitation works can be found from Horizon3 blog post.
Further reading about how to use DPL Architect for security use-cases can be found at Speed up your security investigations with DPL Architect.
How to mitigate
Update to the latest version by applying the latest patches or mitigate by following the instructions within the VMSA-2023-0021.
Sources