Exploit Me, Baby, One More Time: Command Injection in Kubernetes Log Query
Editorial and additional commentary by Tricia Howard
Executive summary
Akamai security researcher Tomer Peled recently discovered a vulnerability in Kubernetes that was assigned CVE-2024-9042.
The vulnerability allows remote code execution (RCE) with SYSTEM privileges on all Windows endpoints within a Kubernetes cluster. To exploit this vulnerability, the cluster must be configured to run the new logging mechanism “Log Query.”
The vulnerability can be triggered with a simple GET request to the remote node.
Successful exploitation of this vulnerability can lead to full takeover on all Windows nodes in a cluster.
This vulnerability can be exploited on default installations of Kubernetes that opted-in to use beta features (earlier than version 1.32.1), and was tested against both on-prem deployments and Azure Kubernetes Service.
In this blog post, we provide a proof-of-concept curl command and discuss possible mitigations.
Introduction
Kubernetes and containers in general have become a predominant force in the security world — and, as such, they’ve been a point of interest for researchers worldwide (including us). Our research journey initially led us to CVE-2023-3676: a command injection vulnerability that could be exploited by applying a malicious YAML file to the cluster. That research led to discovering several other issues in the Kubernetes source code that also allow for complete cluster takeover.
Those vulnerabilities were not all we found — there was also a significant finding in the sidecar project git-sync. We presented those findings at Red Team Village at DEF CON 32, and in preparation for the talk, we stumbled upon the topic for this blog post: Another opportunity for command injection — Kubernetes’ beta feature for their larger logging framework Log Query.
What is Log Query?
Log Query allows users to query remote machines for their system status by using either the CLI or curl. For example, a user can type the following command to query the status of the kubelet service on a remote node:
kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet"
When we saw the example command above, our previous research came to mind: Would the query that is sent to the remote machine be validated?
Figure 1 is an example of the source code for Log Query in which we found several PowerShell commands being built.
Fig. 1: PowerShell commands in Log Query
These commands are used to retrieve logs from a given node based on a number of different parameters, such as Windows service names, time of use pattern, and more.
From previous experience (CVE-2023-3676), we know that Kubernetes does not necessarily validate user input before inserting it as a parameter when building a PowerShell command. We wanted to see if that’s the case with Log Query as well, and found that the service names are actually being validated using a predefined regex (Figure 2).
Fig. 2: Service name validation using regex
The name reServiceNameUnsafeCharacters suggests that this check is for services only. This hypothesis is further strengthened by the fact that there is no other regex present in the parsing process.
After looking at other parameters, it was apparent that, indeed, the only parameter that is being validated is the service name. This means that we can look at other parameters in hopes of achieving a command injection. In Log Query, there are several different optional parameters that users can supply, but the only one that is a string is the Pattern parameter (Figure 3). As we mentioned above, there is no validation over the parameter (apart from its length).
Fig. 3: Building of Pattern parameter
It is worth noting that exploiting this vulnerability was not trivial. We tried triggering the commands on the remote computer by using several different permutations for the PowerShell payload, such as adding brackets, using separators, and so forth, but nothing seemed to work — there were no errors, either, which made figuring this out even more difficult.
The payload is not the problem
After trying many different methods of command injection, we eventually discovered that the problem is not with the payload itself. To exploit this vulnerability, we needed to specify a service that is logging its status natively to event tracing for Windows (ETW) and not to the regular klog framework, as the vulnerable check exists only when logging to ETW.
A Kubernetes environment using Calico (a popular open source network interface for Kubernetes) provides a reliable way to exploit this vulnerability through the Non-Sucking Service Manager (NSSM).
In a Calico setup, NSSM is usually present to control the state of the Kubernetes services. In other environments/setups they use different ways to handle service states.
- Kubernetes doesn't manage NSSM's logging output, ensuring logs are written directly to ETW and not through klog.
Show me an exploit query
This is how an exploit query would look like:
curl "<Kubernetes API Proxy server IP>/api/v1/nodes/<NODE name>/proxy/logs/?query=nssm&pattern=’\$(Start-process cmd)’"
*The authentication token required to talk with the API server is redacted
Sharp-eyed readers may have wondered about the apostrophes used to escape our malicious command. We need them because Kubernetes inserts our input using the following command:
…Where-Object -Property Message -Match '%s'...
Similarly to classic SQL injection attacks, we need to add apostrophes to escape their pattern so our input is parsed as a separate command (Figure 4).
Fig. 4: procmon output of successful command injection
Priority patch
For you to be affected by this vulnerability, you must be using a Kubernetes version earlier than 1.32.1. If you haven’t patched this vulnerability yet, it’s a good idea to prioritize this one. This is especially true for organizations with Windows nodes within a cluster as this is where the vulnerability lies.
Fortunately, this does not seem to be the industry standard. An administrator can easily test whether your organization cluster contains Windows nodes by running the following command on the cluster controller (Figure 5).
Fig. 5: Command to test for Windows nodes in a cluster
Am I vulnerable?
It is easy to determine if you’re vulnerable. Notice the “os=windows” part. If there are not any Windows nodes, the command would have had no output, meaning you’re not vulnerable.
Since this is a beta feature, the cluster also needs to be configured to use the framework itself.
To check whether the feature is enabled, administrators can look at the “feature-gate” startup parameter for “kubelet.” You can get more information on the Kubernetes site.
Patch analysis
To fix this vulnerability, Kubernetes decided to use an environment variable named “kubelet_pattern”, before passing the value to the PowerShell command itself.
That way the user input will be considered as a string literal instead of a subexpression that needs to be evaluated.
Mitigation
To help mitigate this vulnerability, administrators can use the role-based access control (RBAC) module to control who has access to Log Query and even disable access to it entirely. RBAC is a method that segments user operations according to who the user is. For example, each user can only create pods in their own namespace or can only view information for allowed namespaces. This would reduce the risk of RCE not only in execution but also in detection — abnormal user behavior is often a tell-tale sign of mischief.
Keep in mind that this vulnerability only affects Windows nodes. If your Kubernetes cluster doesn’t have any Windows nodes, you are in the clear for this one. However, it is still recommended to keep your patches as up-to-date as possible to avoid other unknown vulnerabilities.
Since the issue lies within the source code, this threat will remain active and exploitation of it will likely increase — this is why we strongly advise patching your cluster even if it doesn’t have any Windows nodes.
Conclusion
In this blog post, we showed how an attacker with query privileges can execute commands on any Windows node inside the cluster. This vulnerability can be exploited by a simple curl command, instead of having to submit a YAML file. This fact imposes a big risk because its exploitation is harder to mitigate and detect. Kubernetes sanitization problems are not unique to Log Query, as we have previously discussed.
Blue team members should be on the lookout for unusual behavior coming into their organizations. This attack vector can lead to a complete cluster takeover; therefore, it's important for us to raise awareness and help security admins know about the potential danger.
The Akamai Security Intelligence Group will continue to research threats such as these and report on them for our customers and the security community at large. For real-time updates on what we’re working on, follow us on X (formerly Twitter.)
We want to thank the Kubernetes team for their response and discourse.
Timeline
06/28/2024 — Vulnerability disclosed to Kubernetes team
07/18/2024 — Kubernetes team began working to fix the issue
07/30/2024 — CVEs assigned by Kubernetes team
01/16/2025 — Kubernetes released the CVE fixes
01/24/2025 — Blog post published