memcached on port 11211 UDP & TCP being exploited

Posted on

 

TLP:WHITE

UPDATE: As of 2018-03-17 ( Morning Update), more attack using the memcached reflection vector have been unleashed on the Internet. As shared by  Akamai Technologies “memcached-fueled 1.3 Tbps Attacks,” the application factors are “Internet Impacting.” Mitigation and Remediation Efforts are reducing the number of potential memcached reflectors. Please keep up the good work.

Operators are asked to port filter (Exploitable Port Filters), rate limits the port 11211 UDP traffic (ingress and egress), and clean up any memcached exposed to the Internet (iptables on UNIX works).  These mitigations should be on IPv4 and IPv6! There is not excuse for ISPs, Telcos, and other operators for not acting. NTT is an example of action. As stated by Job Snijders <job@ntt.net> on the NANOG List:

“NTT too has deployed rate limiters on all external facing interfaces on the GIN backbone – for UDP/11211 traffic – to dampen the negative impact of open memcached instances on peers and customers.

The toxic combination of ‘one spoofed packet can yield multiple reponse packets’ and ‘one small packet can yield a very big response’ makes the
memcached UDP protocol a fine example of double trouble with potential for severe operational impact.”

Will your Efforts Help Safeguard the Internet?

Yes! The proof is in the data. Qrator has been tracking the number of potential memcached reflectors. As see from their data, the list of potential reflectors has been going down. Your efforts to protect your network, your customers and the Internet will work with low-cost techniques which are already built into your network equipment.

(Source: Provided by Qrator)

Recommendations for ISPs, Telcos, Mobile Operators, and Cloud Providers

All Operators and Enterprise Networks – memcached on port 11211 UDP & TCP being exploited. This is now new. We know how reflection attacks work (send a spoofed packet to a device and have it reflected back. This vector has an extremely high amplification rate (ranges vary per test, but all of them are huge). One operator reported one reflection inbound produced +500 Mbps stream outbound. 

Operators are asked to review their networks and consider updating their Exploitable Port Filters (Infrastructure ACLs) to track or block UDP/TCP port 11211 for all ingress and egress traffic.

  • Deploy INGRESS (coming into the ASN) and EGRESS (going out of the ASN) UDP/TCP port filters on 11211. This might have an impact on ephemeral ports for other services. As noted in “Should we deploy Exploited Port Filtering on Ephemeral Ports?” operator deployment experience over the last decade shows minimal customer support complains about the port filtering.  The filters must be IPv4 and IPv6.
     
  • Deploying an explicit IPv4 block on UDP 11211 with a length of 1428 will stop the reflection attack. Don Smith <Donald.Smith@CenturyLink.com> from CenturyLink has worked with his peers in the industry to determine that 1428 is the ‘magic number’ for memcached amplified packet length.  “….block src port UDP:11211 length 1428 and you are going to stop all or nearly all of the reflection/amplification!”
  • Consider Rate Limiting UDP port 11211 to minimize risk. As mentioned by NTT, there are cases where rate limiting on IPv4/IPv6 on UDP port 11211 would be a way to minimize perceived collateral risk with ephemeral ports. 
  • Clean up Internet exposed memcached services. These services can use iptables on UNIX to filter, be removed, or other mitigations applied.  The new version of memcached on GITHUB has disabled UDP by default …. “As reported, UDP amplification attacks have started to use insecure internet-exposed memcached instances. “  Digital Ocean is one of the Operators who had a lot of memcached on their network. They have posted a “how to guide” for how they have been remediating the attack vector: How To Secure Memcached by Reducing Exposure.  This is a good example of community collaboration. 

This white paper provides details on Exploitable Port Filters: http://senki.mystagingwebsite.com/operators-security-toolkit/filtering-exploitable-ports-and-minimizing-risk-to-and-from-your-customers/

Recommendations for Enterprises

Enterprises are asked to update their iACLs, Exploitable Port Filters, and Firewalls to track or block UDP/TCP port 11211 for all ingress and egress traffic. These are the same key recommendations for ISPs (above).Deploying these filters will help protect your network, your organization, your customers, and the Internet. In addition, Enterprise should call their ISPs and ask:

  1. What are you doing to protect against memcached reflection exploits on your network? How are you protecting our network?
  2. Are you deploying Source Address Validate (SAV) and IETF BCP 38 on your network to ensure people cannot use your network for attacks?

Asking Operators drive Internet citizenship to help each other on the Internet. 

Resources Which are Scanning for memcached services

Example of how the Filter can be Deployed

NTT and Job Snijders (job@ntt.net) demonstrate how ISPs, Telcos, and other Operators can work together for the “common Internet.” In an NLNOG 2018-03-01 post  [NLNOG] memcached UDP/11211 IOS XR ratelimit example, Job list out the Cisco IOX XR filters they used. We will be working on more examples. Consider all of these examples as starting points. They provide examples and context to adopt into your router’s configurations. 

Cisco IOS XR Example

This example is from Job Snijders from what NTT deployed. 

I recommend adding memcached UDP/11211 to the same "exploitable ports"
list as NTP, CHARGEN and SSDP. Below is a configuration example for IOS
XR to rate-limit these amplification-sensitive UDP ports to 1% of of the
port's capacity.

    ipv4 access-list exploitable-ports
     permit udp any eq ntp any
     permit udp any eq 1900 any
     permit udp any eq 19 any
     permit udp any eq 11211 any
    !
    ipv6 access-list exploitable-ports-v6
     permit udp any eq ntp any
     permit udp any eq 1900 any
     permit udp any eq 19 any
     permit udp any eq 11211 any
    !
    class-map match-any exploitable-ports
     match access-group ipv4 exploitable-ports
     match access-group ipv6 exploitable-ports-v6
     end-class-map
    !
    policy-map ntt-external-in
     class exploitable-ports
      police rate percent 1
       conform-action transmit
       exceed-action drop
      !
      set precedence 0
      set mpls experimental topmost 0
     !
     class class-default
      set mpls experimental imposition 0
      set precedence 0
     !
     end-policy-map
    !
    interface Bundle-Ether19
     description Customer: the best customer
     service-policy input ntt-external-in
     ipv4 address xxx/x
     ipv6 address yyy/y
     ...
    !
    interface Bundle-Ether20
     service-policy input ntt-external-in
     ...
    ... etc ...

Please share your own examples too!

JUNOS Example

. The following is from Jared Mauch from Akamai that would get people on Juniper routers started.

 term limit-junk {
  from {
    protocol udp;
    source-port [ 123 11211 ];
       }
 then policer police_junk;
  }
   term accept {
   then accept;
  }
...
 policer police_junk {
   if-exceeding {
     bandwidth-limit 1024m;
     burst-size-limit 256k;
    }
 then discard;
 }

Memcached “KillSwitch”

“Kill Switches” are considered “Active Countermeasures. Active Countermeasures is where one connects back to a device they do not own and make any changes. Organizations cannot assume that Active Countermeasures are legal. Every organization considering active countermeasures, “attacking back,” or using tools which “flush a cache,” need to consult with their legal team. At this time, the memcached DOS reflection mitigated and remediated is working. The use of packet filters, rate-limiting, patches, and removal (see above).

There are two cases have referenced memcache active countermeasures:

The industry response team’s investigation considered approaches that would clear violated memcached systems primed for DOS reflection. The technique would reply to known exposed and primed Memcache systems with a “delete <key>.” The method is viable, but not part of the first wave of industry recommendations. The approach contained many unknown back-end risk variables.

As mentioned, the industry ‘active countermeasures’ approaches to DOS attacks malware infections, and other security threats are an “evolving” and considered legal grey areas.  Organizations are consoled seek appropriate legal counsel before considering active countermeasures. 

Resources on memcached Exploit

More information about this attack vector can be found at the following (below). It is strongly encouraged to read through the articles and security advisories. These would help provide context unique to your network and additional details to deploy defenses on your network. 

National CERT, Vendor SIRT, and FIRST Team Advisories

The Forum of Incident Response and Security Team (FIRST) Community works with their constituents to address a range of security issues and incidents. These are the applicable advisories, notices, blogs, and other bulletins on this issue:

Tutorials, Technotes, and Technical Guides

The following are details on how to remediate/mitigate this issue. 

Blogs about Memcached as a DOS Reflection Attack

Blogs tell the story before, during, and after the incident. They share an example of how others experiences and managed the incident. 

 

Need Security Advice?

If you find your organization needs help and worry about the FUD from the industry, reach out and ask for help. You can reach me at bgreene@senki.org. Start with the Operator’s Security Toolkit. It is the no-nonsense security for all Operators. It provides details to help them build more security resilient networks. In the meantime, stay connected to the Senki Community to get updates on new empowerment and security insights.