Your Cache Is Exposed
It didn’t have to be this way. This was never meant to be on the internet.
On February 28, 2018, Akamai recorded a 1.35 Tbps distributed denial-of-service (DDoS) attack against one of our customers. The attack was driven by a relatively new vector, memcached reflection. Possibly the largest publicly disclosed DDoS attack to date, the memcached attack was more than twice the size of the largest DDoS attacks that were launched against an Akamai customer by the operators of the Mirai botnet in September 2016.
Memcached is a service that is meant to cache data and reduce the strain caused by storage intensive tasks and services. In theory, it is only supposed to be used on an internal/private network to assist with short-term storage and delivery of data among systems as part of an application or service. As an example, it finds extensive real-world use as a caching system for production databases, in which the application server will check for cached results before doing more expensive database (and possibly file system) operations.
The problem began when developers, seeking higher performance, decided to add user datagram protocol (UDP) networking support to the memcached project. This helps to reduce networking overhead by speeding up the content delivery process and overall performance of the service. Because memcached was only supposed to run on internal networks where trusted machines could communicate with it, there is no authentication mechanism for the service.
Exposing memcached on the public internet before the inclusion of UDP into the software already presented enough privacy and security concerns to make it unthinkable. The addition of UDP simply added “It can be used as a DDoS reflector” to the list of reasons why the service should never be accessible over the internet.
The primary issue, as it relates to DDoS, isn’t memcached-specific, it’s an issue with almost every software or service that uses UDP. With TCP data flows, there is a three-way handshake that needs to be completed before real communication can begin. UDP doesn’t require this handshake, which allows spoofed requests and in turn “reflection” attacks. Because of this lack of handshake and other mechanisms present in TCP connections, UDP greatly reduces the overhead and, in turn, the speed of the network operations. It makes servicing the requests faster while also reducing resource consumption.
Along with the fact that it’s fast, the default configuration of memcached supports a stored value of up to one megabyte per key. As with other amplification vectors, you send a small packet and get a larger response — but memcached can reflect or amplify substantially more than other vectors. Memcached can have an amplification factor of roughly 500,000 times the initial query, meaning a 203-byte request results in a 100-megabyte response. This massive amplification factor means a seemingly small set of exposed machines can easily produce huge amounts of reflected traffic, far more traffic than is typically seen in these types of attacks. To further complicate matters, defense strategies against volumetric attacks leave a lot to be desired. If you’re not using a mitigation provider or do not have good relationships with your upstream transit providers who can help shed traffic before it hits you, you’re going to have a bad time. The bottom line is you’re going to need fat pipes to ride out this storm.
There has been some rumbling around the internet about a “flush_all kill switch” as a suggested fix. It was even deployed as a feature by a mitigation service/gear provider. The “flush_all” command essentially marks all keys on the memcached server as invalid, allowing the server to purge old data and free up memory. In theory, if an attacker is reflecting payloads off a memcached server using a key-value pair that they’ve setup, sending the “flush_all” command will cause the keys they’re using in the attack traffic to no longer exist from a querying standpoint. In theory, this could reduce the traffic to near zero as the server fails to look up the payload being requested by the attacker.
This reminds me of the Mark Twain quote: “Never argue with a fool; onlookers may not be able to tell the difference.”
There are several reasons why this “solution” is worrisome. First, you’re sending commands to someone else’s server — a server that is already being abused by an unauthorized party. Now you’re abusing it, too, just in a different way, which lands you in a very gray legal area. You may be flushing data from the server that it legitimately needs to power an application or service; you don’t know what that server is being used for or what collateral damage you may inflict. Second, requests that have already been made will continue to serve as proverbial data fire hoses. The “flush_all” will not work because the request happened before the flush, and that data is still being served to the old request. Third, an attacker only needs milliseconds to reset their key, so unless you continuously spam the server with “flush_all” requests, you likely won’t be able to keep up as you’re now engaged in a battle centered on a race condition.
This all seems pretty gloom and doom, but there are some positive thoughts we can take away.
There will always be people who are going to exploit services for as long as they can, but there are also a lot of people behind the scenes who want to see those issues fixed. Thanks to the coordinated efforts of the community, we’ve seen memcached reflection capabilities degraded rapidly and significantly since the initial attacks. At the start, there were approximately 100,000 memcached servers available on the internet. Of those, 52,000 also exposed UDP services capable of being used for DDoS. As of this publication, the estimate is that 43,000 of those UDP memcached services have disappeared. That’s not to say they’re actually gone, but they are no longer publicly reachable because of actions taken on the ISP and server providers’ networks.
Although this is great work by all parties involved, and is very promising for the future of the internet and collaboration across our industry, we’re far from worry-free regarding memcached reflection, as even a few hundred of these servers should be a concern because of the overwhelming amplification factors they’re capable of producing.
At this point, memcached has likely run its course and the clean-up efforts have gone well. However, because of the long tail of unpatched/unprotected servers and their potential to generate large floods of traffic, memcached as a DDoS concern will likely be with us in some form for the foreseeable future.
