What a Cluster: Local Volumes Vulnerability in Kubernetes
Editorial and additional commentary by Tricia Howard
Executive summary
Akamai security researcher Tomer Peled recently discovered a high-severity vulnerability in Kubernetes that was assigned CVE-2023-5528 with a CVSS score of 7.2.
The vulnerability allows remote code execution with SYSTEM privileges on all Windows endpoints within a Kubernetes cluster. To exploit this vulnerability, the attacker needs to apply malicious YAML files on the cluster.
This vulnerability can lead to full takeover on all Windows nodes in a cluster.
This vulnerability can be exploited on default installations of Kubernetes (earlier than version 1.28.4), and was tested against both on-prem deployments and Azure Kubernetes Service.
- In this blog post, we provide a proof-of-concept YAML file as well as an Open Policy Agent (OPA) rule for blocking this vulnerability.
Introduction
Kubernetes and containers in general have become a predominant force in the security world — and, as such, they’ve been a point of interest for researchers worldwide (including us). Our research journey initially led us to CVE-2023-3676 (CVSS of 8.8): a command injection vulnerability that could be exploited by applying a malicious YAML file onto the cluster. Since the Kubernetes framework uses YAML files for basically everything — from configuring the Container Network Interface to pod management and even secret handling — an exploitation of this vulnerability could have disastrous consequences.
The discovery of this vulnerability led to the discovery of two others that share the same root cause: insecure function call and lack of user input sanitization.
The lack of sanitization of the subPath parameter in YAML files that creates pods with volumes opens up an opportunity for a malicious injection. This was the original finding, but at the tail end of that research, we noticed a potential place in the code that looked like it could lead to another command injection vulnerability. After several tries, we managed to achieve a similar outcome: executing commands as the “kubelet" service (SYSTEM privileges). This is where we will start our journey today with CVE-2023-5528.
In this blog post, we will go through the vulnerability details, and the issues in Kubernetes source code that allow it, as well as analyze the Kubernetes team’s patch and its effectiveness. Although patching as quickly as possible is recommended, we’ve included short guides on how to look for affected nodes and how to apply an Open Policy Agent (OPA) rule to help detect and block this kind of behavior.
This post highlights once again how crucial it is to verify Kubernetes configuration YAMLs, since input sanitization is lacking in several code areas in Kubernetes itself and its sidecar projects (like ingress, for example).
Vulnerability details
Before we get into the specifics of this vulnerability itself, we must first understand a few key components within Kubernetes.
What are Kubernetes volumes?
Kubernetes volumes is a feature aimed at supporting the sharing of data between pods, or storing it persistently outside of a pod’s lifecycle. There are many different volume types developers can use. For example, in our previous research on CVE-2023-3676 we used hostPath volumes. For this vulnerability, we are focusing on local volumes, another volume type within Kubernetes. Local volumes are designed to allow users to mount disk partitions inside a pod, while hostPath volumes are designed to allow users to mount directories from their node (host) into a pod.
While creating a pod that includes a local volume, the kubelet service will (eventually) reach the function “MountSensitive()”. Inside it, there’s a cmd line call to “exec.command”, which makes a symlink between the location of the volume on the node and the location inside the pod (Figure 1).
Fig. 1: Vulnerable cmd line
Many terminals use some version of command concatenation (Figure 2) in their operations for ease of use. That is also the case for Windows’ Command Prompt (cmd) — by using the token “&&”, the terminal will execute two or more commands one after the other.
C:\Users\user>echo "by using &&" && echo "we can execute multiple commands in the same command line"
"by using &&"
"we can execute multiple commands in the same command line"
C:\Users\user>
Fig. 2: Command concatenation in cmd
The fact that we can control one of the parameters in the cmd execution means that we can use command injection. There are some prerequisites for it, though — for users to use local volumes, they need to specify or create a persistentVolume.
What are persistentVolumes?
persistentVolumes are storage resources that a cluster admin can create to provision storage space ahead of time that will last beyond the lifetime of the pod (Figure 3). Once a persistentVolume is created, a user can ask for storage space using a persistentVolumeClaim.
Fig. 3: An example of persistentVolume
This is where the injection can be placed. An attacker can change the value of the “local.path” parameter inside the persistentVolume YAML file to add a malicious command that will be executed during the mounting process.
In Figure 4, we have used the benign “&calc.exe&&” (which opens a calculator on the node), but this process can be used for a far more malicious outcome.
Fig. 4: persistentVolume with our “malicious” command
Figure 5 shows how a successful exploitation will look like on a target node after the injection of our “malicious” calculator command.
Fig. 5: Proof of concept of the attack
Patch analysis
In an effort to remove the opportunity for injection, the Kubernetes team chose to delete the cmd call, and replace it with a native GO function that will perform the same operation “os.Symlink()” (Figure 6).
Fig. 6: How the function looks after the patch
Now, the GO “os” library will only perform a symlink operation, as was intended initially.
Am I vulnerable?
For users to be affected by this vulnerability, Kubernetes must be an earlier version than 1.28.4. If you haven’t patched it yet, it’s a good idea to prioritize this one. This is especially true for organizations with Windows nodes within a cluster, as this is where the vulnerability lies.
Fortunately, this does not seem to be the industry standard. An administrator can easily test whether your organization cluster contains Windows nodes by running the command shown in Figure 7 on the cluster controller.
root@controller:~/$ kubectl get nodes -o wide --show-labels | grep “os=windows”
akswin000000 Ready agent 4d17h v1.26.6 agentpool=win,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=windows…
akswin000001 Ready agent 4d17h v1.26.6 agentpool=win,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=windows…
root@controller:~/$
Fig. 7: The command that shows windows nodes in a cluster
It is easy to determine if you’re vulnerable. Notice the “os=windows” part. If there are not any Windows nodes, the command would have had no output, meaning you’re not vulnerable.
Mitigation
The only mitigation available is to patch Kubernetes to a version later than 1.28.3.
That being said, we know that immediate patching is not something that can be done in some organizations and networks. To help accommodate the risk of not patching, we’ve provided an OPA rule to help detect and block this kind of behavior.
package kubernetes.admission
deny[msg] {
input.request.kind.kind == "PersistentVolume"
path := input.request.object.spec.local.path
contains(path, "&")
msg := sprintf("malicious path: %v was found", [path])
}
OPA is an open-source agent that allows users to receive data on traffic going in and out of nodes and take policy-based actions on the received data.
Keep in mind that this vulnerability only affects Windows nodes. If your Kubernetes cluster doesn’t have any Windows nodes, you don’t have to rush to patch this specific vulnerability. But it’s important to patch it anyway when you have the time.
Since the issue lies within the source code, this threat will remain active and exploitation of it will likely increase — this is why we strongly advise patching your cluster even if it doesn’t have any Windows nodes.
Conclusion
This vulnerability is a great example of why the shared responsibility model is crucial in security. Being aware of the lack of input sanitization in Kubernetes source code means you can take outside precautions to help avoid a serious security impact.
Seven different command injection vulnerabilities were discovered in 2023 alone, with more opportunities in other areas of the code. Blue teams and their organizations should be more alert to this growing trend and try to monitor YAML file contents since they may contain hidden threats. The OPA rule we provided in this post can help with that effort.
The Akamai Security Intelligence Group will continue to monitor this threat and others like it and publish our findings. To keep up with this vulnerability and other security research, you can follow us on X (formerly Twitter).
We want to thank the Kubernetes team for their quick response and smooth communication.
Disclosure timeline
11/01/2023 — Vulnerability disclosed to Kubernetes team
11/11/2023 — CVEs assigned by Kubernetes team
11/14/2023 — Kubernetes released the CVE fixes
03/13/2024 — This blog post published