A Vulnerability in Debian’s Apt Allows for Easy Lateral Movement in Data Centers
A new vulnerability in Debian’s Advanced Package Tool (apt) is the latest big tool in the data center attacker’s arsenal. The vulnerability (CVE-2019-3462) is in Debian’s high-level package management system, which is used by system administrators to install, upgrade and remove software packages. The vulnerability can be exploited when administrators install or upgrade software package on vulnerable servers.
The apt package management software is part of every Debian based Linux distribution, covering Debian, Ubuntu and a whole group of smaller distributions such as Kali, TailsOS and many others. Distrowatch lists over 100 active distributions (large and small) based on Debian. All of these are likely to be vulnerable.
Using simple network traffic spoofing and traffic injection techniques, an attacker can easily impersonate an apt server and exploit this vulnerability to bypass security measures and further spread in the network towards the organization’s crown jewels. If exploited, the vulnerability may allow an attacker to run code at root permissions on victim servers and basically perform any action they want, such as installing trojans, ransomware, worms and more.
In this post we will cover some of the technical details of the vulnerability, explain how it can be exploited by an attacker in data centers, and discuss effective ways to mitigate against such attacks.
Lateral movement through apt
On January 22nd, Max Justicz published a write up detailing a vulnerability in the apt client. Using Man in the Middle techniques, an attacker can intercept the apt communication while it downloads a software package, replace the requested package content with their own binary, and execute it with root privileges.
The vulnerability has probably existed since at least version 0.7.21 (HTTP redirect support added in January 2009) and has been fixed in apt as part of version 1.4.9
(Debian security announcement 4371) and Ubuntu security notice 3863.
The attack relies on two principal flaws
1 – The apt http downloader allows for content injection
2 – By default, apt downloads software packages over HTTP rather than HTTPS
The first flaw allows an attacker to control the behavior of the apt package manager, resulting in an arbitrary code execution. But to reach a situation where an attacker can control content, he must be able to inject himself into the communication between apt and remote servers. This is possible because by default, apt communicates over HTTP (and not HTTPS).
Exploiting this inside a data center network requires traffic redirection capabilities. To execute a traffic redirection maneuver, an attacker will need either to control a privileged location (such as a proxy or router) or to somehow spoof one of the update servers, by abusing a variety of network injection attacks, such as DNS poisoning.
Once the attacker is able to intercept apt traffic, they must wait to see a server update. In many organizations, this happens on a regular basis. Once they capture apt traffic, they can reply with malicious content. From this stage, they can write files that will be parsed by apt , and using different parsing mistakes, cause apt to execute a crafted file with root permissions.
This vulnerability is an excellent opportunity for an attacker to further propagate inside a data center. In many networks, east-west traffic is not monitored and is easy to spoof.
An apt takeover
When the apt process requests a software package, it communicates with a repository server using a spawned worker process such as /usr/lib/apt/methods/http or /usr/lib/apt/methods/ftp. These processes are responsible for handling the nitty gritty details of the specific protocol.
The communication between these worker processes and the apt process is made through a text-based protocol via stdinstdout interface. This interface contains status messages (downloading, waiting, etc.) and control messages (download this, redirect here). Messages are formatted in a similar way to HTTP messages.
Code Text
Key: Value
Key2: Value
For example,
601 Configuration
Config-Item: APT::Architecture=amd64
Config-Item: APT::Build-Essential::=build-essential
Config-Item: APT::Install-Recommends=1
(…many more lines omitted…)
This example and all following examples are taken from Max Justicz’s excellent write up.
The vulnerability resides inside the http worker. If the worker receives an HTTP redirect reply from the server , the new URI provided by the server is sent to the apt process as is, allowing a malformed URI to inject content.
So while a valid redirection would look like the following text
URI: https://deb.debian.org/debian/pool/main/c/cowsay/cowsay_3.03+dfsg2-3_all.deb
New-URI: https://deb.debian.org/debian/new/folder/directory/new_file_name.deb
A malicious reply can look like this:
URI: https://deb.debian.org/debian/pool/main/c/cowsay/cowsay_3.03+dfsg2-3_all.deb
New-URI: https://deb.debian.org/new-uri
Foo: Bar
In this case, the attacker is redirected to a new URI but more importantly the `Foo: Bar` key-value was injected by adding a new line character to the end of the redirection URI.
An attacker can exploit this to inject messages to the communication channel between the apt and its worker. Due to the textual fashion of the protocol between these two processes, the attacker can inject new messages after the New-URI value. The result is that apt will read two messages. One, informing apt that the request was redirected and an additional attacker generated message. This message is under attacker control and he can write that the download request was completed. This can be seen in the following example:
103 Redirect
URI: https://deb.debian.org/debian/pool/main/c/cowsay/cowsay_3.03+dfsg2-3_all.deb
New-URI: https://deb.debian.org/payload
201 URI Done
URI: https://deb.debian.org/payload
Filename: /var/lib/apt/lists/deb.debian.org_debian_dists_stretch_Release.gpg
Size: 20070
Last-Modified: Tue, 07 Mar 2017 00:29:01 +0000
MD5-Hash: 27967ddb76b2c394a0714480b7072ab3
MD5Sum-Hash: 27967ddb76b2c394a0714480b7072ab3
SHA256-Hash: 858d5116a60ba2acef9f30e08c057ab18b1bd6df5ca61c233b6b7492fbf6b831
Checksum-FileSize-Hash: 20070
What happened is that the attacker not only injected new values but by knowing the precise protocol was able to inject new messages. Following the above malicious reply, the apt process will send a request for the new URI in response to the 103 Redirect message. Then before waiting for a new reply the apt process will read the injected 201 URI Done message and assume the download is complete.
Note that to be able to properly inject content, the attacker must match his attack to the exact version of apt running on the victim, but this can be achieved through trial and error.
The apt protocol
To abuse this vulnerability and run code on a victim machine, we need to take a step back and understand the communication between the client and a repository.
We’ll break down the apt-get update command. When the apt-get update is run, we are requesting the apt package manager to update its list of packages that exist on remote repositories. A typical install has multiple remote repositories to use (saved in /etc/apt/sources.list) and for each one, there is a manifest that lists what packages are available in that repository.
The manifest is stored in a file named Release – a simple text file, listing package files available in the repository with their hash.
For example, a snippet from the Release file for the Debian 9 repository looks like this:
Origin: Debian
Label: Debian-Security
Suite: stable
Version: 9
…
MD5Sum:
d41d8cd98f00b204e9800998ecf8427e 0 contrib/Contents-amd64
4a4dd3598707603b3f76a2378a4504aa 20 contrib/Contents-amd64.gz
…
SHA1:
da39a3ee5e6b4b0d3255bfef95601890afd80709 0 contrib/Contents-amd64
a0fddd5458378c1bf3c10dd2f5c060d1347741ed 20 contrib/Contents-amd64.gz
…
Inside each file listed in Release, there are more file listings, with more hashes, that detail specific files for specific package (deb) files.
“These checksums allow apt to verify that it has downloaded a correct copy of the Packages file, with a checksum that matches the one in the Release file. And when it downloads an individual package, it can also check its checksum against the content of the Packages file. If apt fails at either of these steps, it will abort.” (Taken from the SecureApt entry in the Debian wiki)
From a security point of view, this makes the release file a single point of failure. If an attacker can modify the release file, they can easily feed malicious package files that look legitimate.
The solution to this problem is an additional file named Release.gpg that contains a textual GPG signature. Using the signature stored in this file, apt checks if Release is valid. With this process, the Release file can be validated.
Code execution
The attack flow – from MitM to code execution
Using this data and the content injection vulnerability, an attacker can write and utilize different files to gain code execution:
- The client runs apt update requesting a Release and Release.gpg files from the repository server. In return the attacker provides a malicious Release.gpg file which is a perfectly valid file but with additional data. The gpg file is then stored by apt in a path known to the attacker.
- Next, the client runs apt install which leads to a request for a deb package. Then, using the content injection vulnerability, the attacker injects a message directing apt that the deb package was downloaded pointing to the location of the Release.gpg file.
At this point, the apt package manager will parse the Release.gpg file, as if it was a deb package and execute its instructions under root privileges. This “trick” works for two reasons. The deb file format cares about the file content at the start of the file and ignores any appended content. The second is that the gpg tool is very permissive in what it accepts, as long as somewhere in the file the signature appears. The result is an attacker that can construct a file that looks like the following image.
The malicious Release.gpg serves double duty as a PGP signature and a debian file
Fixing a ten year old vulnerability
The fix for the vulnerability is simple and can be seen in git repository of apt (commit 5eb01ec1).
The fix changes the code to ensure that the new URI contains only alphanumeric text and other printable characters. This prevents an attacker from causing content injection or any other manipulation.
Mitigation
The obvious solution to this vulnerability is to securely patch the apt package. This is not trivial because we typically upgrade packages using apt, and the vulnerability is in the upgrade process run by apt! Use the following command line
The command line apt -o Acquire::http::AllowRedirect=false can be used as an alias for apt as a workaround to prevent redirections, according to Debian this may cause security updates to fail and therefore is not a long-term solution.
An alternative solution is to switch apt to use HTTPS , but not all mirrors support HTTPS and this can have a significant bandwidth cost in large networks.
An even better solution will include preventing these types of attacks all together. Apt traffic should only go to well-known machines, either mirrors inside a network or a specific list of machines in the internet. To enforce this, you might want to create a rule that limits traffic originating from apt and its sub processes to specific IPs. This makes sure that an attacker cannot simply rely on redirecting apt traffic to any server controlled by them. This can be achieved with Guardicore Centra, which can easily detect traffic from apt and make sure it goes only to allowed servers (internal or external).
In this image from Centra, we see apt-get communicating with an unknown internal server
This defense method is applicable not only for apt – it should be applied to every critical IT infrastructure in your network, such as DHCP.
An apt concern
This is not the first vulnerability in apt, a similar attack in 2016 was discovered and vulnerabilities have also been found in the Windows Update mechanism. We can assume that more bugs reside in these more than a decade old code bases. Network security professionals should assume update mechanisms can be abused and make sure to harden core IT.
