Git-Syncing into Trouble: Exploring Command Injection Flaws in Kubernetes
Executive summary
Akamai researcher Tomer Peled found a design flaw in Kubernetes’ sidecar project git-sync that allows for potential command injection. He’ll present these findings at DEF CON 2024.
This design flaw can cause either data exfiltration of any file in the pod (including service account tokens) or command execution with the git_sync user privileges.
To exploit the flaw, all an attacker needs to do is apply a YAML file on the cluster, which is a low-privilege operation.
This vulnerability can be exploited on default installations of Kubernetes on all platforms (including Amazon Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), and Google Kubernetes Engine (GKE), and Linode).
In this blog post, we provide a proof-of-concept (PoC) YAML file, as well as demos that showcase the potential impact.
Since a CVE has not been assigned to this design flaw, there is no patch for it — it remains exploitable.
The Kubernetes team has encouraged us to share this research to raise awareness about these attack vectors.
Introduction
Kubernetes is no stranger to command injection vulnerabilities. In 2023 alone, 7 such vulnerabilities were found, including several we found ourselves. Input sanitization concerns drove us to look more deeply into ancillary attack vectors: Is this attack vector unique to Kubernetes’ main project or is it more widespread? There are several sidecar projects associated with Kubernetes in which vulnerabilities can hide, including git-sync.
The git-sync project is meant to connect a pod and a git repository to sync changes between their site/server automatically instead of making changes manually through a CI/CD solution. For example, users could use this feature to link their nginx pod with a repository containing the files they want to expose through an nginx pod.
In this blog post, we will go through the details of the design flaw, the issues in the Kubernetes source code that allow it, and some mitigations. We will also discuss Kubernetes’ response to our findings.
Attack vector details
By looking into the git-sync use page, we can see that it supports many possible configuration parameters so that a user can customize git-sync to their needs. This allows for a potentially large attack surface that an attacker could exploit (Figure 1).
Fig. 1: Some of git-sync’s parameters
The Kubernetes framework uses YAML files for basically everything — from configuring the Container Network Interface to pod management and even secret handling, so a vulnerability within YAML can be quite dire. In this case, a pod can be created to use git-sync to connect to a remote repository (or an attacker).
Figure 2 is an example of a configuration YAML file that deploys a pod with git-sync.
Fig. 2: Configuration YAML file that deploys a pod with git-sync
The two parameters that stood out the most as potential attack vectors were GITSYNC_GIT and GITSYNC_PASSWORD.
Per the official git-sync documents, GITSYNC_PASSWORD_FILE is “The file from which the password or personal access token (see github docs) to use for git authentication (see --username) will be read.“ This hints at a possibility for data exfiltration of the “accesstoken” or other files on the pod.
GITSYNC_GIT is (again, from the official docs) “The git command to run (subject to PATH search, mostly for testing). This defaults to "git," which means we can choose a binary that will be executed instead of git, and use it for code execution on the cluster.
Proposed attack chain
With the above information in mind, we set out to prove our theories. We believe there are a few attack vectors that attackers can exploit.
Stealthy code execution
An attacker with low privileges (Create privileges) on the cluster or namespace can apply a malicious YAML file containing a path to their binary, causing it to be executed under the git-sync name.
The binary file needs to be inside the pod, which can be done in a few different ways, such as via Kubernetes probes, Kubernetes volumes, or LOLBins that come with the git-sync pod (Figure 3).
Fig. 3: YAML example using LOLBins
After applying the YAML file, in the eyes of blue team personnel git-sync is the one communicating with a remote server, and therefore it's reasonable to assume it would be trusted to communicate externally. This allows attackers to bypass security measures as a bonus to the command execution.
Figure 4 is a POC of a potential attack starting an XMrig cryptominer under the git-sync user.
Fig. 4: PoC of potential XMRig cryptominer
Now, when a network administrator audits existing pods and their communication outside the company network, they will most likely see the git-sync user communicating with an outside server.
Data exfiltration
An attacker with Edit privileges can edit a git-sync deployment to change or add the GITSYNC_PASSWORD_FILE parameter and point it at any file on the pod. This will cause git-sync to send over the file as a means of authentication on its next connection to the git repository.
If the attacker also modifies the git repository location, and sets up a pseudo repository server, the next deployment of the git-sync process inside the pod will send the file in the GITSYNC_PASSWORD_FILE parameter from the pod to their machine. There are no restrictions on the file paths or permissions required for the GITSYNC_PASSWORD_FILE.
A high-risk exfiltration is not hard to imagine. Consider the following: Attackers can use this technique to retrieve the access token of the pod, which would allow them to interact with the cluster under the guise of the git-sync pod (Figure 5).
Fig. 5: PoC of potential GITSYNC_PASSWORD_FILE attack
Disclosure and Kubernetes’ response
We originally disclosed our findings to the Kubernetes team back in December 2023. After some discussion with the Kubernetes team, it was decided that editing YAMLs is considered a high-privilege operation so our findings don’t cross a security threshold. From our perspective, however, although edit operations on a pod are considered privileged, lateral movement is still possible with this technique. There was also a concern of losing integrity: The attacker would be able to steal information as if they were a legitimate git-sync user.
On the matter of GITSYNC_GIT, the Kubernetes team acknowledged that the privileges required for this type of action are low, but did not think that the low-privilege operations would lead to any harmful behavior. However, we believe that the design flaw we’ve described would allow attackers to execute commands while spoofing their identity, and the only bar to harmful behavior is low privileges on the cluster. This attack flow is especially dangerous in organizations that have pre-authorized git-sync communication in their cluster.
In both cases, Kubernetes encouraged us to share this very valuable research online as it “helps remind admins to think carefully about the surface area they expose to users.”
Mitigations
As the attack techniques described in this blog post are not considered vulnerabilities by the Kubernetes team, there will not be a patch for them. Therefore, to reduce the attack surface these “features” allow, we recommend a few potential mitigations.
First, increase your monitoring of communication that leaves the organization, specifically from Kubernetes pods. Apply extra care to git-sync pods if that granularity is possible.
Second, we recommend auditing git-sync pods to see what command they are running. This can be done with the following command:
kubectl describe pod <git-sync-pod>
For example, when executing this command on the cryptominer shown in Figure 4, we can see the –git argument that would have raised a red flag upon detection, especially when the value of the argument is not “git” (Figure 6).
Fig. 6: Executing this auditing command on the cryptominer in Figure 4
This audit is a good general security practice, so we recommend executing this command on all pods, not just git-sync pods.
We’ve also written an OPA rule that can detect one of our proposed attack vectors, in which attacks replace the git binary with a malicious payload:
package kubernetes.admission
deny[msg] {
input.request.kind.kind == "<Deployment/Pod>"
path := input.request.object.spec.env.name
contains(path, "GITSYNC_GIT")
msg := sprintf("Gitsync binary parameter detected, possible payload alteration, verify new binary ", [path])
}
Conclusion
In this blog post, we showed two attack vectors in the git-sync Kubernetes sidecar project. Both vectors are due to a lack of input sanitization, which highlights the need for a robust defense regarding user input sanitization. These Kubernetes sanitization problems are not unique to git-sync, as we have previously discussed.
Blue team members should be on the lookout for unusual behavior coming from the gitsync user in their organizations. We believe these attack vectors can have a very large impact on companies; therefore, it's important for us to raise awareness and help security admins know about the potential danger.
The Akamai Security Intelligence Group will continue to research threats such as these and report on them for our customers and the security community at large. For real-time updates on what we’re working on, follow us on X (formerly Twitter.)
Despite our disagreement on the outcome, we want to thank the Kubernetes team for their response and discourse.