Can't Be Contained: Finding a Command Injection Vulnerability in Kubernetes
Editorial and additional commentary by Tricia Howard
Executive summary
Akamai security researcher Tomer Peled recently discovered a high-severity vulnerability in Kubernetes that was assigned CVE-2023-3676 with a CVSS score of 8.8.
This discovery led to the identification of two more vulnerabilities as they share the same root cause: insecure function call and lack of user input sanitization.
The vulnerability allows remote code execution with SYSTEM privileges on all Windows endpoints within a Kubernetes cluster. To exploit this vulnerability, the attacker needs to apply a malicious YAML file on the cluster.
This vulnerability can be exploited on default installations of Kubernetes, and was tested against both on-prem deployments and Azure Kubernetes Service.
We provide a proof-of-concept YAML file as well as an OPA rule for blocking this vulnerability.
Introduction
YAML (which stands for YAML Ain't Markup Language) is a data serialization language similar to JSON which is mainly used in configuration files. As such, it plays a big role in Kubernetes, the newly ubiquitous container orchestration system that helps organizations automate the deployment, scaling, and management of containerized applications. The Kubernetes framework uses YAML files for basically everything — from configuring the Container Network Interface to pod management and even secret handling.
YAML files in Kubernetes have been the subjects of research in recent years. Considering Kubernetes relies on YAML for cluster configuration, this research topic is particularly interesting to dive into.
How have YAML and Kubernetes been exploited previously?
In 2022, CVE-2022-1471 was found inside the constructor of SnakeYAML, a known parser of YAML files. The vulnerability allowed the creation of unsafe objects, which could lead to execution of code on vulnerable applications that use it. Another example that showcases the possible impact of YAML files can be seen in this CVE advisory about CVE-2021-25749, written by the Kubernetes security team. This vulnerability allows attackers to bypass the verification of who is allowed to run as root by misspelling their name in the YAML file.
As part of our research into Kubernetes we came across CVE-2017-1002101 and CVE-2021-25741. These vulnerabilities demonstrated how attackers can use race conditions and symlinks in conjunction with the subPath subproperty in a YAML file to gain access to privileged data outside of the container.
These past vulnerabilities inspired us to pursue the direction of path processing in the Kubernetes codebase, which eventually led to the discovery of the vulnerability we are discussing today.
In this blog post, we will demonstrate how an attacker with “apply” privileges — the privileges required to interact with the Kubernetes API — can inject code that will be executed on remote Windows machines with SYSTEM privileges.
Vulnerability details
When a pod is created, the user has the option to create a shared directory between the pod and the host. This feature — called volumes — is useful, for example, while serving websites through a container in a pod. We may want to share the site files from a directory that is shared with the host; this way, we would be able to change the site layout as we’d like and wouldn't need to recompile an image containing the site each time.
Enabling volumes is done by including the volume parameter in the pod’s YAML file. The locations for where to mount the volume are listed in mountPath (the location in the container) and hostPath (the location on the host).
Most important for us, we can mount our shared directory or file in a chosen location using the subPath subproperty. All relevant properties for volumes can be seen in Figure 1.
Fig. 1: YAML volume configuration
The parsing of our YAML file is performed by kubelet — a central service in Kubernetes that is responsible for running the containerized applications on a node. As part of the patch for CVE-2021-25741, kubelet validates every parameter in the YAML file, and also ensures that no symlinks are created as a result of using the subPath parameter by calling the inner function “isLinkPath”. The function is displayed in Figure 2.
Fig. 2: subPath function to check symlink paths
The function takes as a parameter the subPath that was supplied by the user in the YAML file. It then uses this path to create a PowerShell command meant to determine the path type (i.e., whether it’s a symlink or not). The formatted PowerShell command is then immediately invoked by the “exec.Command” function call.
The presence of “exec.Command” combined with unsanitized user-supplied input strongly hints at a command injection opportunity.
PowerShell allows users to evaluate values inside strings before they are used. This can be done by adding $(<experssion_to_be_evaluated>) to your string, for example (Figure 3).
PS> echo “the value of 1+1 is $(1+1)
Output:
the value of 1+1 is 2
Fig. 3: Example of PowerShell evaluating expressions inside strings
This example is pretty simple, but in fact any PowerShell command can be inserted between the parentheses and will be evaluated — such as $(Start-Process cmd), $(Invoke-Expression exp), and other PowerShell treats.
An attacker can abuse this subPath evaluation to reach the vulnerable code and execute any command they want with SYSTEM privileges (kubelet’s own context) from remote nodes, and gain control over all Windows nodes in the cluster. Figure 4 shows a malicious subPath value.
Fig. 4: YAML file with command injection inside the subPath parameter
Proof of concept showcasing CVE-2023-3676
The proof of concept is simply a YAML file that contains the evaluation of the PowerShell command. You can find this file, as well as a video showcasing the terminal being started, in our GitHub repository.
This CVE led to the discovery and fix of more command injection vulnerabilities. They were collectively given the numbers CVE-2023-3955 and CVE-2023-3893.
Patch analysis
The Kubernetes team chose to patch this class of vulnerabilities by passing parameters from environment variables instead of from user input (Figure 5). By passing down the values in this manner, the parameters are treated as strings — therefore, they will not be evaluated as expressions by PowerShell.
Fig. 5: Patched function for CVE-2023-3676
Am I vulnerable?
As we mentioned above, all Kubernetes versions below 1.28 are vulnerable to this CVE. There are several methods to determine whether one of the nodes in your cluster is a Windows node. One method is to execute the command in Figure 6.
kubectl get nodes -o wide --show-labels | grep “os=windows” Output: akswin000000 Ready agent 4d17h v1.26.6 agentpool=win,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=windows… akswin000001 Ready agent 4d17h v1.26.6 agentpool=win,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=windows… |
Fig. 6: A kubectl command to determine if one of the nodes in your cluster is a Windows node
Mitigation
Patching is the most reliable way to protect yourself from this vulnerability. In cases in which patching is not an option, we have outlined a number of potential ways to protect against this vulnerability, and you can choose the one that suits your environment best.
Previous solutions
For CVE-2023-3676, Kubernetes admins can disable the use of Volume.Subpath. This will make sure that your cluster is safe against this vulnerability, but will disable a mechanism that is sometimes important for clusters in production.
OPA
Open Policy Agent (OPA) is an open source agent that allows users to receive data on traffic going in and out of nodes and take policy-based actions on the received data. OPA uses Rego as the language for their engine, with which admins can create rules to block certain YAML files from being implemented. Figure 7 illustrates a rule that will deny creation of pods with a malicious subPath.
package kubernetes.admission
deny[msg] {
input.request.kind.kind == "Pod"
path := input.request.object.spec.containers.volumeMounts.subPath
not startswith(path, "$(")
msg := sprintf("malicious path: %v was found", [path])
}
Fig. 7: A rule that will block the creation of pods with a malicious subPath
RBAC
Role-based access control (RBAC) is a method that segments user operations according to who the user is. For example, each user can only create pods in their own namespace or can only view information for allowed namespaces. This may help limit the number of users who can perform actions on a cluster and help focus attention on the ones who do have the relevant privileges for exploitation.
Conclusion
CVE-2023-3676 requires low privileges and, therefore, sets a low bar for attackers: All they need to have is access to a node and apply privileges. As we discussed in this blog post, successful exploitation of this vulnerability will lead to remote code execution on any Windows node on the machine with SYSTEM privileges.
High impact coupled with ease of exploitation usually means that there is a higher chance of seeing this attack (and similar attacks) on organizations. In fact, the only limiting factor with this vulnerability is its scope — it is restricted to Windows nodes, which are not very popular today.
It is important that admins patch their Kubernetes clusters to the latest version available to prevent exploitation of these vulnerabilities. If patching is not available to you immediately, we recommend that you use one of the methods described above to help defend against this vulnerability.
We want to thank the Kubernetes team for their very quick response and smooth communication.
Disclosure timeline
07/13/2023 — Vulnerability disclosed to Kubernetes team.
07/19/2023 — CVEs assigned by Kubernetes team
08/23/2023 — Kubernetes released the CVE fixes
09/13/2023 — blog post published
