On February 28, 2018, the popular GitHub’s code hosting website was hit by the largest-ever distributed denial of service (DDoS) attack.
The DDoS attack peaked at record 1.35 Tbps by abusing the memcached protocol to power so-called memcached DDoS attacks.
Memcached is a free and open source, high-performance, distributed memory caching system designed to speed up dynamic web applications by alleviating database load.
Clients communicate with memcached servers via TCP or UDP on port 11211.
Researchers from Cloudflare, Arbor Networks and security firm Qihoo 360 discovered that recently attackers are abusing the memcached for DDoS amplification attacks.
Chinese experts warned about abuses of memcached DDoS attacks in November.
The abuse of memcached servers in DDoS Attacks is quite simple, the attacker sends a request to the targeted server on port 11211 spoofing the IP address of the victim. In a memcached DDoS attack, the request sent to the server is composed of a few bytes, while the response can be tens of thousands of times bigger, resulting in an amplification attack.
Experts at Cloudflare dubbed this type of attack Memcrashed, according to the researcher the amplification technique could allow attackers to obtain an amplification factor of 51,200.
The Github website is protected by the anti-DDoS service provided by the firm Akamai that confirmed the impressive magnitude of the attack that hit its client.
“At 17:28 GMT, February 28th, Akamai experienced a 1.3 Tbps DDoS attack against one of our customers, a software development company, driven by memcached reflection. This attack was the largest attack seen to date by Akamai, more than twice the size of the September, 2016 attacks that announced the Mirai botnet and possibly the largest DDoS attack publicly disclosed.” reads the analysis published by Akamai.
“Because of memcached reflection capabilities, it is highly likely that this record attack will not be the biggest for long.”
According to GitHub, the attack was widespread, it originated from over a thousand different autonomous systems (ASNs) across tens of thousands of unique endpoints.
“On Wednesday, February 28, 2018 GitHub.com was unavailable from 17:21 to 17:26 UTC and intermittently unavailable from 17:26 to 17:30 UTC due to a distributed denial-of-service (DDoS) attack.” states an advisory post published by GitHub.
“Between 17:21 and 17:30 UTC on February 28th we identified and mitigated a significant volumetric DDoS attack.
The attack originated from over a thousand different autonomous systems (ASNs) across tens of thousands of unique endpoints. It was an amplification attack using the memcached-based approach described above that peaked at 1.35Tbps via 126.9 million packets per second.”
Github routed the traffic to Akamai service to mitigate the ongoing DDoS attack.
“Given the increase in inbound transit bandwidth to over 100Gbps in one of our facilities, the decision was made to move traffic to Akamai, who could help provide additional edge network capacity. At 17:26 UTC the command was initiated via our ChatOps tooling to withdraw BGP announcements over transit providers and announce AS36459 exclusively over our links to Akamai.” continues Github.
“Routes reconverged in the next few minutes and access control lists mitigated the attack at their border. Monitoring of transit bandwidth levels and load balancer response codes indicated a full recovery at 17:30 UTC. At 17:34 UTC routes to internet exchanges were withdrawn as a follow-up to shift an additional 40Gbps away from our edge.”
GitHub confirmed that the first portion of the attack peaked at 1.35Tbps, while a second part peaked 400Gbps after 18:00 UTC.
Github said it plans to expand its edge network and mitigate new attack vectors.
Researchers believe that threat actors in the wild will abuse misconfigured Memcached servers in future attacks, unfortunately, many of them are still exposed on the Internet.
Cloudflare recommends disabling UDP support unless it’s needed and isolating memcached servers from the Internet. Internet service providers have to fix vulnerable protocols and prevent IP spoofing.
“Internet Service Providers – In order to defeat such attacks in future, we need to fix vulnerable protocols and also IP spoofing. As long as IP spoofing is permissible on the internet, we’ll be in trouble.” concluded Cloudflare.
“Developers – Please please please: Stop using UDP. If you must, please don’t enable it by default. If you do not know what an amplification attack is I hereby forbid you from ever typing
SOCK_DGRAM into your editor.”
(Security Affairs – memcached DDoS Attacks, Github)