Living off the VPN — Exploring VPN Post-Exploitation Techniques
Executive summary
In this blog post, Akamai researchers highlight the overlooked threat of VPN post-exploitation; that is, we address techniques that can be used by threat actors after compromising a VPN server to further escalate their intrusion.
Our findings include several vulnerabilities that affected Ivanti Connect Secure and FortiGate VPNs.
In addition to the vulnerabilities, we detail a set of no-fix techniques that can affect the Ivanti Connect Secure and FortiGate products, and potentially other VPN servers, as well.
Our research shows that, in many cases, a compromised VPN server could allow attackers to easily gain control over other critical assets in the network.
This blog post aims to raise awareness of these risks, and presents best practices that should be followed by defenders to minimize the risks of VPN post-exploitation techniques.
Introduction
We’ve all heard this story before: A critical vulnerability is discovered in a VPN server. It's exploited in the wild. Administrators rush to patch. Panic spreads across social media.
Last year was pretty rough for VPN security — it seemed like every other month a critical vulnerability was patched or discovered after exploitation in the wild by threat actors. Although this type of activity saw a significant increase in 2023, it is not new. Attackers have long sought to exploit VPN servers because they are accessible from the internet, expose a rich attack surface, and are often lacking in security and monitoring.
Historically, VPN servers have been primarily abused to achieve a single objective — initial access. Attackers would compromise the internet-facing VPN server and use it as a beachhead into the internal network, enabling them to conduct their intrusions.
Although this approach is very effective, we asked ourselves: Should control over a VPN server be seen solely as a gateway to the network?
We wanted to see what else might be possible.
The following blog post explores VPN post-exploitation techniques that can be used by an attacker who has already compromised a VPN server by other means (vulnerabilities, stolen credentials, etc.) to achieve additional goals.
Living off the VPN
The main approach that threat actors use to perform post-exploitation techniques is to target the OS of the device. After gaining remote code execution (RCE), an attacker can drop a custom implant on the device OS. From this point, the threat actor can control every aspect of the VPN; for example, they can hook functions to leak sensitive information, attempt to evade detection by manipulating logs, or modify system configuration to maintain persistence on the device.
While this approach has many advantages, it comes with one main drawback — it's expensive. Since appliances usually run on a custom hardened OS, the amount of effort required to develop and maintain a custom implant for a VPN device can be significant. This means that VPN post-exploitation techniques were typically only used by top-tier nation-state threat actors.
Exploring a different approach
We decided to explore a different approach — an “easier” form of VPN post-exploitation. Instead of relying on a custom implant running on the OS, we decided to abuse the existing functionality of the device. We asked ourselves: What can an attacker accomplish while using only the VPN management interface?
This approach, which we dubbed “living off the VPN,” has at least two advantages.
This type of access can be easier to obtain than full RCE — access to the management interface can be obtained through an authentication bypass vulnerability, weak credentials, or phishing.
This approach can be more cost-effective, as we avoid the effort of developing a custom payload.
As our test subjects, we chose to target two of the leading VPN servers on the market — Ivanti Connect Secure and FortiGate. We uncovered 2 CVEs, and a set of no-fix techniques that can be used by attackers who control the VPN server to take over other critical assets in the network, which can potentially turn a VPN compromise into a full network compromise.
Although our findings focus on FortiGate and Ivanti, we believe that variations of the techniques we’ve found are likely to be relevant for additional VPN servers and edge devices.
Abusing remote authentication servers
Some of the most interesting assets handled by a VPN server are external credentials. Although VPN servers support using local users for authentication, an external authentication server is used in many cases.
Instead of maintaining a separate set of credentials per user, admins can choose to use an existing identity provider to authenticate their users. The user sends their “normal” credentials to the VPN server, which are validated via the remote authentication server (Figure 1).
Fig. 1: Using a remote authentication server to authenticate users
Several types of authentication servers can be used for this purpose — the 2 main options are the LDAP and RADIUS.
We identified a few techniques that can allow attackers to capitalize on this behavior to compromise these external credentials, which potentially provides attackers with access to additional resources in the network.
Intercepting LDAP credentials
One very popular authentication server option for VPNs is LDAP, most commonly an Active Directory (AD) domain controller. With this configuration, users can access the VPN via their domain credentials, making this a very convenient choice.
When configuring an LDAP authentication server, credentials of an AD service account are provided to allow the VPN to query user information. An example of this configuration in FortiGate can be seen in Figure 2.
Fig. 2: LDAP Authentication server configuration in the FortiGate GUI
When a user attempts to authenticate to FortiGate or Ivanti by using LDAP credentials, the VPN verifies them by contacting the LDAP server. The exact implementation varies slightly between the two, so let's examine both.
FortiGate LDAP authentication
To validate the credentials, FortiGate attempts to use them to authenticate to the remote server. This process is performed using simple authentication — meaning that FortiGate sends the password to the server in cleartext (Figure 3).
Fig. 3: LDAP simple bind exposes the cleartext user password
Two sets of credentials are transmitted during this process: the credentials of the LDAP service account configured on FortiGate, and the credentials supplied by the authenticating user. Both are sent in cleartext.
While the default configuration is LDAP, FortiGate also supports LDAPS and TLS, which should prevent cleartext communication.
Ivanti LDAP authentication
Ivanti’s implementation for LDAP authentication is slightly different, and supports two types of LDAP authentication servers: normal LDAP and Active Directory (Figure 4).
Fig. 4: Ivanti authentication server options, including LDAP and Active Directory servers
LDAP authentication server
With an LDAP authentication server, the process is pretty similar to FortiGate’s process; that is, when plain LDAP is used, a simple bind is performed and the AD service account and authenticating user credentials are both transmitted in cleartext (Figure 5).
Fig. 5: Ivanti transmitting LDAP credentials in cleartext
However, when configuring an LDAP authentication server, the default setting is TLS, and LDAPS is also supported. Therefore, Ivanti instances are less likely to use plain LDAP.
AD authentication server
The second type of LDAP authentication server that can be configured is an AD server. When using this option, authentication is performed using Kerberos, meaning that no passwords are transmitted (Figure 6).
Fig. 6: Secure LDAP authentication using Kerberos
Capturing cleartext LDAP credentials
For both FortiGate and Ivanti, when plain LDAP is used to authenticate a user, any credential sent to the VPN can potentially be compromised. This compromise can be carried out by an attacker that sits between the VPN and the LDAP server, or an attacker that controls the VPN server.
How can we capture the credentials without dropping an implant on the device? Well, luckily for us, FortiGate and Ivanti both include a built-in packet capture feature. By using it to capture LDAP packets, an attacker can intercept all the credentials that go through the VPN (Figure 7).
Fig. 7: Using FortiGate’s or Ivanti’s packet capture feature to capture LDAP communications
In cases where a secure protocol is used, no cleartext credentials are sent from the server. Despite that, an attacker with control over the VPN can still trivially capture them. If we control the VPN, there’s nothing to stop us from modifying the configuration and downgrading it back to plain LDAP.
For normal LDAP servers, we can simply change the configuration to plain LDAP. To downgrade Ivanti’s AD server, we can use the AD connection details to configure a new, normal LDAP server instead of the existing one. In both cases, this change should be transparent to users and will cause the VPN to send the passwords in cleartext.
The bottom line is this: If LDAP authentication is used, regardless of the configuration, an attacker who compromises the VPN will be able to easily obtain credentials and pivot into the domain.
Registering a rogue authentication server
As we’ve mentioned, when authenticating a remote user, the VPN will contact the appropriate authentication server to validate the provided credentials. We identified a method that abuses this authentication flow to compromise any credential provided by a user to the VPN.
This technique works by registering a rogue authentication server that will be used by the VPN when authenticating users. To help you understand how this can be implemented, we will first describe some details of the authentication process on both VPNs.
FortiGate user groups
FortiGate users can be granted different permissions, and have different policies applied to them. Instead of managing each user individually, users can be included in user groups; that is, sets of users that can have policies and permissions applied to them.
When configuring such a group, we can include users from remote groups — groups that are present on a remote authentication server. For example, we can include users from a specific AD group (Figure 8).
Fig. 8: FortiGate user group configuration
We observed an interesting behavior that occurs when using remote groups. Let’s say that we configure a user group that includes two entities: a local FortiGate user and a remote group present on an LDAP server.
With regard to authentication, we would expect the following behavior:
When the local member of the group authenticates to FortiGate, their credentials will be verified using FortiGate’s local user database.
When a member of the remote group authenticates to FortiGate, their credentials will be verified using the remote group LDAP server.
As it turns out, however, that’s not actually the case. After we add a remote group to a user group, whenever any member of the user group authenticates, FortiGate will attempt to validate their credentials using both the local user database and the remote authentication server. Moreover, if we add more than one remote server to a user group, FortiGate will attempt to authenticate users using all the configured authentication servers.
Essentially, FortiGate doesn’t match a specific user with its appropriate authentication method, it simply attempts all of them. If any succeed, the user is authenticated.
Ivanti authentication realms
Ivanti implements a similar feature to user groups called authentication realms. Unlike FortiGate’s user groups, each Ivanti authentication realm is limited to a single authentication server. To manage users from different servers, separate realms must be used.
Despite this restriction (and conveniently for us), Ivanti allows us to configure an “additional authentication server” (Figure 9). This option is meant to enable certain SSO configurations.
Fig. 9: Configuring an additional authentication server for an Ivanti user realm
When an additional authentication server is configured, Ivanti will attempt to validate the credentials using both servers. Authentication will succeed only if both servers approve it.
Creating a rogue authentication server to leak credentials
An attacker can abuse these behaviors to leak the credentials of any user that authenticates to FortiGate or Ivanti, using any type of remote authentication server. By adding a rogue authentication server to a user group or realm, we can cause the VPN server to leak credentials to an attacker-controlled server (Figure 10).
Fig. 10: Adding a rogue authentication server to compromise client credentials
These credentials could be of clients connecting to the VPN or for administrators connecting to the management interface. Any authentication that is managed using a user group or realm could be hijacked using this method.
To implement this on FortiGate, we created a remote group that uses our rogue server, and then added it to our target user group. With Ivanti, we simply added our rogue server as an additional authentication server for our target realm.
Now, whenever a member of the target user group/realm authenticates, the VPN will attempt to validate their credentials using our server (Figure 11).
Fig. 11: Adding a rogue RADIUS authentication server to a user group
To capture the credentials, we will use a RADIUS authentication server. RADIUS authentication is convenient in this scenario for two reasons:
Credentials are sent to the server during the initial request without first verifying whether the user exists on the server.
Credentials are sent to the server encrypted with a key that is determined by the attacker, enabling them to recover the cleartext credentials (Figure 12).
Fig. 12: An encrypted password in a RADIUS authentication message
The following script (based on RFC 2865) can decrypt RADIUS passwords:
import hashlib
# Authenticator value can be obtained from the PCAP
authenticator = bytearray.fromhex("98f245b6e3724f5873fa74e576323c67")
enc = bytearray.fromhex("15644ca055b817ce683083fecebc2902016f2890660d0dfcaee214a0dbaa1046")
# Pre-shared key configured by the attacker when setting up the rogue server
secret = bytes("verygoodpsk", "utf-8")
chunk_len = 16
enc_chunks = []
dec = ""
for i in range(0, len(enc), chunk_len):
enc_chunks.append(enc[i:i+chunk_len])
for chunk in enc_chunks:
dec_chunk = b""
chunk_key = hashlib.md5(secret + authenticator ).digest()
i = 0
for enc_byte in chunk:
dec_chunk += (enc_byte ^ chunk_key[i]).to_bytes(1,"big")
dec += chr(enc_byte ^ chunk_key[i])
i+=1
authenticator = chunk
print(dec)
One final note: As we’ve mentioned, when Ivanti uses an additional authentication server, both servers need to approve the user for the authentication to succeed. If our server doesn’t approve the supplied credentials, users will be unable to authenticate to Ivanti. To prevent this, we need to make sure that our RADIUS server approves any credentials that are provided to it, which can easily be configured.
Extracting configuration file secrets
What may be our most concerning finding involves VPN configuration files.
Among the various interesting settings we can locate in configuration files, one stands out — secrets. VPNs store many secrets in their configuration — local user passwords, SSH keys, certificates, and, most interestingly, credentials of third-party service accounts.
These sensitive files can be obtained by an attacker in two main ways:
After gaining control over a VPN, an attacker can export the device configuration through the management interface.
An attacker can locate a previously exported configuration file that was saved to a public location, such as a shared folder.
To protect them, secrets are stored in the configuration file in an encrypted form. Figure 13 shows an example of an encrypted secret in a FortiGate configuration file.
Fig. 13: An encrypted password inside a FortiGate configuration file
Now, let’s decrypt some secrets!
Decrypting secrets from a FortiGate configuration file
One might think that the secrets are hashed and therefore cannot be recovered, but that can't actually be the case. Take integration passwords, for example — as the cleartext password is required to connect to the third party, it has to be stored using reversible encryption.
This was proven in a blog post by Bart Dopheide that dug into the FortiGate password encryption process. FortiGate uses AES to encrypt all the secrets in the configuration. What key is used to perform this encryption? Dopheide found that a single hard-coded key is used across all FortiGate appliances. This key could not be changed. The original blog post doesn’t share it, but a quick Google search will lead you right to it.
Fortinet assigned CVE 2019–6693 to this issue, and implemented a fix by allowing users to change the hard-coded key to a custom one.
Even after this fix, the problem is still very relevant today. The key was not changed, so by default, FortiGate appliances still use the same key. This means that if an attacker were to obtain a configuration file of a FortiGate appliance with the default configuration, they will be able to decrypt all the secrets stored on the device. This code implements the decryption process.
Okay, but what if the FortiGate admin followed the best practice and used a custom key instead of the default one? We discovered that if we control the VPN, we can still easily obtain the secrets.
The private-data-encryption setting is used to control the custom encryption key. What's surprising is that admins can simply disable this feature. This requires no knowledge of the currently configured key, and will revert the encryption of all secrets back to the original hard-coded key.
Reverting the encryption
To revert the encryption, we can use the following FortiGate command-line interface commands:
FGT # config system global
FGT (global) # set private-data-encryption disable
FGT (global) # end
At this point, we can download the configuration file and decrypt the secrets by using the known default key (exactly as we showed before).
Many interesting secrets can be compromised using this method. FortiGate supports integrations with various applications via the “external connector” feature. These connectors serve various purposes, but most of them share an important aspect — they require credentials for the application (Figure 14).
Fig. 14: FortiGate external connector options
This means that FortiGate may contain credentials for critical services such as cloud providers, SAP, Kubernetes, ESXi, and more.
In some cases, the credentials require high privileges for the respective application. For example, the “Poll Active Directory Server” integration requires the credentials of an account with administrative access to a domain controller, potentially turning a FortiGate breach to a full domain compromise immediately.
Decrypting secrets from an Ivanti configuration file
The situation is pretty similar with Ivanti. In fact, it may be worse.
Ivanti also stores secrets using reversible encryption. If we extract the code from the device and dig into it, we will quickly find the secret value that is used as the encryption key, which is (again) a static string (Figure 15).
Fig. 15: Ivanti static encryption key
We have redacted the full key, but by inspecting the beginning of it, we can assume that it has likely not changed since at least 2015, when Juniper last owned Connect Secure.
Ivanti seems to have put a greater effort into protecting the stored secrets, implementing a more complex encryption algorithm than FortiGate. Despite that, this process is obviously still reversible. This means that secrets across all Ivanti instances are encrypted using one static key that cannot be changed. Attackers could use this knowledge to decrypt any Ivanti configuration file.
Letting Ivanti decrypt passwords for us
Our initial approach was to reverse engineer the encryption process to create a decryptor. This is certainly possible, but after a few long hours of looking at decompiled code, we decided to go with a different approach as a proof of concept. We let Ivanti do the heavy lifting for us.
The main idea is to use a dedicated attacker-owned Ivanti instance, “feed” it the encrypted password, and get it to decrypt it for us (Figure 16).
Fig. 16: The decryption process of an Ivanti password in an attacker lab environment
This decryption can be done via the following steps:
Obtain an encrypted password from an Ivanti configuration file
In a lab environment, set up an Ivanti instance and an LDAP server (for example, a Windows server running Active Directory)
On the lab Ivanti instance, configure our LDAP server as an authentication server that uses simple authentication
Export the lab Ivanti configuration to a file
Replace the existing encrypted LDAP password in the configuration file with the encrypted password we are attempting to decrypt
Import the modified configuration file back into the Ivanti instance
Now, when we trigger an authentication attempt to the LDAP server, Ivanti will decrypt the password from the configuration and send it over cleartext to the LDAP server. By capturing the packets, we can obtain the decrypted password (Figure 17).
Fig. 17: Ivanti decrypting our supplied password and leaking the result over LDAP
We use the LDAP server authentication as an “interface” with the decryption process, but it is important to note that we are not limited to LDAP passwords. Our testing showed that this technique will work for all secrets stored in the configuration file, including (but not limited to) AD passwords, RADIUS pre-shared keys, and API keys.
Cleartext MDM server passwords
Ivanti supports a few popular mobile device management (MDM) servers as authentication methods (Figure 18).
Fig. 18: Ivanti MDM server configuration
Like other authentication servers, this integration requires credentials for the MDM server. If we inspect the configuration for the MDM server, we will encounter two interesting fields: “password-encrypted” and “password-cleartext”. And, despite what the names might suggest, both the fields are stored in cleartext (Figure 19). 🤷♂️
Fig. 19: Ivanti MDM configuration containing cleartext passwords
VPN post-exploitation techniques in the wild
It’s likely that some of the techniques we’ve highlighted in this post are already being used in the wild. In their Cutting Edge report, which covered a series of exploitation campaigns against Ivanti appliances, Mandiant researchers shared that attackers were able to compromise the LDAP service account configured on the Ivanti device (Figure 20).
Fig. 20: Mandiant’s report mentioning a compromised LDAP account
Although the Mandiant report doesn’t go into detail regarding how the attackers were able to accomplish this, we believe it is fairly likely that the attackers were able to obtain the credentials using one of the methods we highlighted in this blog; that is, either by extracting them from the configuration file or by sniffing network traffic.
These types of techniques are trivial to implement, and we believe that attackers of all sophistication levels will be able to use them.
Recommendations when using a VPN server
We recommend four principles that should be followed when using a VPN server to minimize the risks from the techniques we just highlighted:
Employ Zero Trust Network Access
Limit service account permissions
Use dedicated identities for VPN authentication
Monitor configuration changes
1. Employ Zero Trust Network Access
One of the main problems with traditional VPNs is their “all or nothing” approach in granting access to the network — users are either “in” (and have complete access to the network) or they’re “out,” (and can’t access anything).
Both of these options are problematic. On one hand, we must provide users with remote access to internal applications. On the other hand, we don’t want an attacker to obtain full access to the network if they were to compromise a VPN server.
Zero Trust Network Access (ZTNA) offers a solution. By defining network access policies per entity, we can allow users to perform approved remote operations, while reducing the impact of a potential breach.
Akamai Enterprise Application Access is a ZTNA solution that offers precise access, based on identity and context, to private applications. It uses identity-based policies and real-time data such as user location, time, and device security to ensure that users only have access to the applications they need and eliminates network level access. It works seamlessly with Akamai MFA to provide strong user authentication.
2. Limit service account permissions
As we illustrated in this post, it is trivial to recover the cleartext passwords of service accounts stored on VPN servers. There is no real way to avoid this, as VPNs require using the cleartext passwords in some cases.
To reduce the impact of a potential VPN compromise, we recommend the use of service accounts with a limited set of permissions — preferably read only. Defenders should try to understand how an attacker could leverage the credentials stored on the VPN, and make sure that a VPN compromise will not lead to a compromise of other critical assets.
Some integrations require a service account with strong permissions for their operation. If possible, they should be avoided.
3. Use dedicated identities for VPN authentication
Now we know that it is trivial for an attacker with control over the VPN to compromise the credentials of authenticating VPN users. Although it could be tempting to use existing authentication services, such as AD, to authenticate users to the VPN, we recommend that you avoid doing that. Attackers with control over the VPN will be able to obtain credentials and use them to pivot into internal assets, turning the VPN into a single point of failure.
We recommend instead that you use a separate, dedicated way to authenticate users to the VPN. For example, perform certificate-based authentication using certificates issued specifically for this purpose.
4. Monitor configuration changes
The techniques that we’ve highlighted will all reflect in the device configuration in one way or another. As network device configuration doesn’t tend to change frequently, these changes can provide a significant detection opportunity.
To identify configuration changes, we recommend periodically pulling the device configuration and comparing it with a “golden image.” Most devices also include internal logging capabilities for system and security events. We recommend that you collect and analyze any available logs and use them to identify suspicious changes to the device configuration.
These four principles all boil down to this: Don’t blindly trust your VPN.
Summary
Attackers are after your VPN. That's a fact. Defenders should assume that it's only a matter of time before a threat actor gains access to their VPN, and they should act accordingly now. Defenders must make sure that a compromised VPN cannot lead to a fully compromised network.
Disclosure timeline
Ivanti
03.26.2024 — Initial report to vendor
04.05.2024 — Ivanti acknowledges the report and confirm they are working on a fix
06.03.2024 — Initial notice to Ivanti that we plan to publish the research on August 7
06.27.2024 — Ivanti states they are working on a fix but cannot confirm the issues will be fixed prior to the publication date
07.08.2024 — Ivanti assigns CVE-2024-37374 and CVE-2024-37375 to the hard-coded key issue, and MDM cleartext passwords, but have not yet released a patch
07.15.2024 — Additional notice to Ivanti about the planned release schedule
08.07.2024 — Research is published
Fortinet
03.22.2024 — Initial report to vendor
04.17.2024 — Fortinet initial response, concluding that none of the described issues require a fix
04.21.2024 — Initial notice to Fortinet that we plan to publish the research on August 7
04.30.2024 — Fortinet informs us they intend to fix the custom encryption key bypass we described
07.15.2024 — Additional notice to Fortinet about the planned release schedule
07.23.2024 — Fortinet states that they will likely not be able to release the patch prior to the publication date
08.02.2024 — Fortinet informed us that after additional consideration, they decided to not fix the custom encryption key bypass as it “does not cross a security boundary”
08.07.2024 — Research is published
