Scanning Akamai’s Edge Servers for Vulnerabilities, Correctly
Continuous monitoring of the Akamai Edge Platform for security vulnerabilities is an integral part of all engineering efforts at Akamai. In addition to our internal vulnerability management program, we engage with third-party assessors to periodically perform external scans of our systems since this is required for compliance with security standards such as PCI DSS and FedRAMP.
We make summary results of these activities available to our customers; however, customers also often request to perform vulnerability scans themselves, either in order to test the security of their web properties served over the Akamai Edge Platform, or sometimes to test Akamai's infrastructure itself. Akamai has established procedures to assist with such assessments, and can accomodate most requests within a framework that avoids negative impact to the rest of our customers. Many customers take advantage of this, and run scans to meet their own compliance goals.
Unfortunately, once customer-performed scans are complete, they are sometimes left with a less-than-ideal report. In reality, however, many (and sometimes all) findings in the results are false positives. This is a familiar situation for Akamai's InfoSec team and a natural outcome of the architectural and operational decisions Akamai has made for its platform. Conducting external vulnerability scans correctly requires proper configuration of the tools used, and then careful interpretation of their output.
The focus of this post is two recurring issues that invariably lead to false positives in customer scans and symptoms that can help customers recognize and address them.
Bad fingerprinting of Operating Systems and applications
Operating system (OS) and application fingerprinting are well-established techniques used in vulnerability scanning. In a nutshell, scanners probe their targets and look for characteristics that can help identify the OS on the machine or the applications listening on open ports, as accurately as possible. Common techniques range from basic "tricks" such as banner grabbing and error message scraping, to complex analyses of the system's TCP/IP stack behavior. (Readers interested in learning more about specific fingerprinting techniques and algorithms can find a myriad of resources on the subject via a simple Internet search; the nmap documentation https://nmap.org/book/osdetect.html is a great place to start.) Scanners use these fingerprints to look up known vulnerabilities for the detected environment and report their findings.
How does this apply to the Akamai Edge Platform?
Akamai delivers content to end users via Edge Servers. Regardless of the specific Akamai service being used, user requests will be served by an Edge Server positioned to offer optimal performance. Edge Servers run our heavily-customized, minimized, and hardened Linux distribution called Akamai Linux Server Install (ALSI), which in turn runs a kernel developed to meet our particular performance and security requirements. Requests that reach Edge Servers are served by Ghost, Akamai's home-grown HTTP server.
When customers perform an external scan of their "Akamaized" website, the scanner hits the Edge Server first. Since ALSI and Ghost are unique to Akamai and exhibit peculiar behavior, many off-the-shelf scanners will fail to fingerprint them correctly and subsequently erroneously classify them as vulnerable.
For example, reports we receive repeatedly misidentify Ghost as outdated, vulnerable versions of Apache Tomcat or Wordpress; of course, we do not run these on Edge Servers. Similarly, ALSI is often fingerprinted as running an archaic Linux 2.6 kernel, with a long list of historical CVEs attached to it (e.g., CVE-2011-3188, https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-3188 is a recurrent false positive finding). In one amusing instance, the Akamai Edge Platform was detected as a network of Internet cameras; and no, we do not operate an Orwellian surveillance platform.
Bad requests to Edge servers
The Akamai Edge Platform is a massively-distributed reverse proxy shared by all customers, and the DNS ecosystem is an integral part of its operation. When customers sign up for Akamai services, they delegate domain name resolution for their web properties to Akamai's authoritative nameservers. So, when an end user attempts to resolve a corresponding name, Akamai will return the IP address of an Edge Server.
This name resolution process is designed to ensure that end users are dynamically mapped to an Edge Server that can offer them optimal performance at that time. However, this also ties into back-end mechanisms that allow an Edge Server to identify the intended destination of traffic it receives. For instance, when TLS is involved, the Edge Server may need this additional information to decide which customer's TLS certificate should be presented to a connecting client. Similarly, Edge Servers may need to inspect HTTP request headers to determine how to correctly enforce customer configurations.
As this overview illustrates, end users first need to perform domain name resolution to locate an appropriate Edge Server and then craft their HTTP requests with correct headers to reach the intended destination. A typical web browser would do exactly that. However, vulnerability scanners often take shortcuts that can function in some circumstances, but not over the Akamai Edge Platform.
The most common issues we encounter stem from scanning methodologies that perform domain name resolution in batches before a scan, resulting in a pool of arbitrary Edge Server IP addresses. Scanners then perform their tests on this pool at a later time. This will violate both requirements we set out before; the Edge Server will not have been properly mapped to expect that connection, and the HTTP requests will not contain the necessary headers for Akamai to process them. As a result, GHost will serve back an error message, and scanners will misinterpret this situation in unpredictable ways, even though a connection to the customer origin was never truly established.
This problem frequently manifests itself in scan results as warnings about Akamai's use of old TLS versions, weak cipher suites, and various certificate errors. Note that customers have full control over how they configure their TLS-fronted Akamai services via Luna Portal. However, next to industry standard options, Akamai also supports deprecated configurations for customers that decide to make a trade-off between security and usability, for example, when they need to support clients with outdated TLS implementations. When vulnerability scanners skip name resolution and attempt to connect to an arbitrary Edge Server IP address that may no longer be mapped for serving the requested content, unexpected behavior may be exposed to the scanner, leading to confusing results. Of course, under normal operation, once the Edge Server determines the correct destination and proceeds to establish a connection, it would enforce the target customers' configuration and respect their TLS parameters. If customers configure their web properties securely, an insecure connection will never be possible.
How to scan Akamai correctly?
Unfortunately, there is no straightforward answer. The Akamai Edge Platform is a complex technology, and performing a correct and effective vulnerability scan requires a high-level working knowledge of the platform, appropriate configuration of the scanning tools, and careful interpretation of results.
We hope that this article will shed light into what to look out for, and help our customers recognize symptoms of inappropriate scan methodologies early on in their efforts. In addition, Akamai's Professional Services team is happy to help customers interpret scan results and give insights into what information is reported. That being said, we ask that customers provide us with as many technical details as possible when requesting our assistance. These may include the target domain names, URLs, or IP addresses scanned, tools used, all scan parameters, time of the scan, and detailed findings. Without this information, it is difficult for us to conduct a detailed investigation.
Food for thought: what to scan?
Akamai's secure delivery platform and security products, such as Kona Site Defender, are designed to block attacks before they reach customers' origin servers. As a result, Akamai helps customers in their security compliance efforts, and external scans of the platform will allow customers to demonstrate adherence to security standards when under audit.
We stress that Akamai's security features are designed to help prevent potential vulnerabilities in customers' origins from being exposed to the outside world and exploited, in essence hiding vulnerabilities. This is by design and necessity. Security products cannot fix customer vulnerabilities; only in-house security engineers can do that. Customers that seek to improve their overall security stance, identify and track their own vulnerabilities, and strive to make their applications secure by design may consider performing internal scans of their origins directly, bypassing the protections afforded by Akamai, so that all security issues can be fully exposed and addressed.
