Recovering Plaintext Passwords from Azure Virtual Machines
Azure provides an incredible amount of add-on services for its IaaS offerings. These services are provided by the Azure Guest Agent through a large collection of plugins such as Chef integration, a Jenkins interface, a diagnostic channel for apps running inside the machine and many more. While researching the Azure Guest Agent, we’ve uncovered several security issues which have all been reported to Microsoft. This post will focus on a security design flaw in the VM Access plugin that may enable a cross platform attack impacting every machine type provided by Azure.
Simply put, attackers can recover plaintext administrator passwords from machines by abusing the VM Access plugin. These may be reused to access different services, machines and databases. Keeping plaintext credentials can also violate key compliance regulations such as PCI-DSS and others. Microsoft has dismissed this issue and replied as follows : “…the way the agent & extension handlers work is by design.”
In this post, we’ll cover the technical details of how the VM Access plugin works and how we managed to abuse this design flaw to recover “secure”plaintext data. Mitigation for this issue will also be discussed.
The Azure VM access plugin
One of the many plugins Azure provides is VM Access, a plugin that helps recover access to machines that users have been inadvertently locked out from accessing. This can be the result of mistakes while configuring the machine or because the login credentials have been forgotten. Azure allows users to reset both the machine configuration and the password of any local user.
Due to the sensitive nature of credentials handling, we decided to take a closer look at this plugin. Password credentials are a valuable target for attackers, as credential reuse is a constant thorn in many organizations security posture. In recent Windows releases, credential storage has been repeatedly hardened and since Windows 8.1 and Windows Server 2012, Windows hasn’t been keeping plaintext passwords at all. Plaintext passwords are even more valuable for attackers as even minor manipulations on passwords can open multiple doors.
We’ve discovered that since late 2015, attackers have been able to recover plaintext credentials after breaching and take over an Azure virtual machine if the VM Access plugin was used at any point on that machine.
Let’s take a quick look at the Azure Guest Agent and assemble the pieces to understand how the plaintext password can be extracted.
How the Azure guest agent works
Azure offers a rich plugin system enabling developers and administrators to interact with virtual machines. The core of this functionality is a cross platform agent, built into every marketplace that provides virtual machine images. This agent is a background service that continually communicates with a controller (part of the Azure infrastructure), receives tasks to execute, performs them and reports back.
At the highest level, the administrator, using the Azure portal or API, provides an operation to be executed on the guest virtual machine. The Azure infrastructure then provides the Guest Agent the requested configuration and if required, the plugin to execute it. For example, if the administrator wants to collect specific diagnostics from a machine, Azure provides a plugin package named IaaSDiagnostics and a configuration file with the parameters encoded inside.
Plugin configuration files
The communication between the controller and the Guest Agent occurs over plain HTTP and an XML based protocol. Each configuration is transmitted as a JSON data structure wrapped in XML and only the raw JSON is stored on disk. The JSON format is identical across all the extensions and is similar to the following data (taken from an VMAccess configuration, version 2.4.2) –
This data is saved in: C:PackagesPlugins<PluginName>RuntimeSettings
This folder is readable by any user but writable only by administrators. For example, the configuration file above was saved in :
C:PackagesPluginsMicrosoft.Compute.VMAccessAgent2.4.2RuntimeSettings.settings
From our perspective, the important part of this configuration file is the sensitive data (such as passwords), which is encrypted and encoded as a Base64 string and stored in protectedSettings.
The sensitive data is encrypted using a certificate generated for communication between the Guest Agent and Azure, given to Azure by the Guest Agent. To read the data encrypted by Azure, the plugin decodes the Base64 data and using the certificate specified by the thumbprint, decrypts the content. On Windows this certificate is saved in the registry under HKLM/Software/Microsoft/SystemCertificates accessible only to administrators and the operating system itself.
Recovering plaintext data
Now we have all the ingredients needed to recover plaintext data. The new password, provided by the administrator through the Azure portal, is saved in the protectedSettings section in the configuration file that was passed to the Guest Agent running on the guest machine. This file is saved to disk containing the new password as a plaintext string first encrypted then base64 encoded.
Once an attacker breaches a machine and successfully escalates privileges, any certificate stored on the machine can be accessed. At that point, all that’s required is reading the base64 encrypted data, finding the certificate using the supplied thumbprint and ta-da – the data is successfully recovered. Here’s some simple C# code to decode this data using available Microsoft APIs:
Compiling this code with a small wrapping will easily recover passwords in Windows. In Linux the vulnerability is the same and the data can be extracted with a few shell commands running as root.
Successful execution on a Linux machine
For an attacker to successfully execute this attack, two conditions are required:
- The VM Access plugin must be used to reset a user’s password.
- A method to read the certificate file using root/administrator permissions.
While privilege escalation may not be trivial, reports about escalation methods are published from time to time. For example, nearly a year ago Microsoft closed a trivial privilege escalation in the Azure Guest Agent on Windows that could give attackers full control of the victim’s system. We’ll have more on this in our next post.
Bypassing defenses as if they don’t exist
A key effort in the past few years is ensuring that even if attackers compromise a machine, they cannot easily recover passwords or use any stored credentials. This is crucial as one of the most common methods to propagate across networks is stealing and reusing credentials. Every operating system provides this guarantee in a different fashion, but at bare minimum this consists of storing only password hashes. In addition, storing plain passwords is forbidden by many compliance standards, such as the PCI and SWIFT.
When it comes to Windows protection, Microsoft has put great efforts into hardening credential storage in Windows and thwarting credential stealing and other attacks – such as Pass the Hash, weaponised by Mimikatz. Over multiple Windows releases, the password storage has been hardened, culminating in Credential Guard – introduced in Windows 10.
With the VM Access plugin, none of this hardening is relevant.
Diagnosis and mitigation
Based on the code provided above, we wrote a diagnostic tool (code available on GitHub, binary available here) to help check which plaintext credentials are stored on an Azure Windows machine. The tool checks for existing VM Access configuration files and if any exists, it displays the recovered credentials. Run the program inside an Administrator command prompt like in the following example:
Result of a successful recovery
If you’ve ever used the VM Access plugin, we recommend that you consider the password to be compromised and change it, without using the plugin. If possible, avoid using the VM Access plugin to reset VM passwords.
If you still prefer using the plugin we suggest deleting the configuration files under
C:PackagesPluginsMicrosoft.Compute.VmAccessAgentRuntimeSettings
after the plugin has finished running, to minimise the time the passwords are written to disk. A sample powershell command could be
rm -Recurse -Force C:PackagesPluginsMicrosoft.Compute.VmAccessAgent2.4.2RuntimeSettings*.settings
The equivalent shell command in Linux is
sudo rm -f /var/lib/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1/config
Summary
Using this design flaw, an attacker can bypass modern security controls quite easily. An attacker with privileged access to a locked down Windows Server 2016 machine with Credential Guard installed can acquire the plaintext password of an administrator user within a few seconds. This is made possible simply because the Password Recovery mechanism was used, a mechanism delivered by none other than… Azure. An attacker can then attempt to move laterally across the network through password reuse (a common problem with many organizations) and take over additional services.
While this attack requires high privileges, privilege escalation vulnerabilities are routinely discovered. We will show an attack leveraging the Azure Guest Agent in a future post.
Lastly, our only reply to Microsoft’s claim of “by design” is why store passwords using reversible encryption in 2018?
