Usually when you think about Denial of Service attacks nowadays, most people think up images of the Anonymous kids running their copy of LOIC in a hivemind or Russian Gangsters building a botnet to run an online protection racket.  Now there is a new-ish type of attack technique floating around which I believe will become more important over the next year or two: the slow http attacks.

Refs:

• Slowloris
• Slow HTTP POST
• RU Dead Yet (RUDY)
• Tor’s Hammer
• XerXes

How Slow DOS Works

Webservers run an interesting version of process management.  When you start an Apache server, it starts a master process that spawns a number of listener processes (or threads) as defined by StartServers (5-10 is a good starting number).  Each listener serves a number of requests, defined by MaxRequestsPerChild (1000 is a good number here), and then dies to be replaced by another process/thread by the master server.  This is done so that if there are any applications that leak memory, they won’t hang.  As more requests are received, more processes/threads are spawned up to the MaxClients setting.  MaxClients is designed to throttle the number of processes so that Apache doesn’t forkbomb and the OS become unmanageable because it’s thrashing to swap.  There are also some rules for weaning off idle processes but those are immaterial to what we’re trying to do today.

Go read my previous post on Apache tuning and stress testing for the background on server pool management.

What happens in a slow DOS is that the attack tools sends an HTTP request that never finishes.  As a result, each listener process never finishes its quota of MaxRequestsPerChild so that it can die.  By sending a small amount of never-complete requests, Apache gladly spawns new processes/threads up to MaxClients at which point it fails to answer requests and the site is DOS’ed.  The higher the rate of listener process turnover, the faster the server stops answering requests.  For a poorly tuned webserver configuration with MaxClients set too high, the server starts thrashing to swap before it hits MaxClients and to top it off, the server is unresponsive even to ssh connections and needs a hard boot.

The beauty of this is that the theoretical minimum number of requests to make a server hang for a well-tuned Apache is equal to MaxClients.  This attack can also take out web boundary devices: reverse proxies, Web Application Firewalls, Load Balancers, Content Switches, and anything else that receives HTTP(S).

Post photo by Salim Virji.

Advantages to Slow DOS Attacks

There are a couple of reasons why slow DOS tools are getting research and development this year and I see them growing in popularity.

• Speed and Simplicity:  Slow DOS attacks are quick to take down a server.  One attacker can take down a website without trying to build a botnet or cooordinate attack times and targets with 3000 college students and young professionals.
• TOR:  With volume-based attacks like the Low Orbit Ion Cannon, it doesn’t make sense to route attack traffic through TOR.  TOR adds latency, throttles the amount of requests that the attacker can send, and might eventually fail before the target’s network does.  Using TOR keeps the defender from tracking you back to your real location.
• Server Logging: Because the request is never completed, most servers don’t make a log.  This makes it very hard to detect or troubleshoot which means it takes longer to mitigate.  I’m interested in exceptions if you know specifics on which webserver/tool combinations make webtraffic logs.
• IDS Evasion: Most DOS tools are volume-based attack.  There are IDS rules to detect these: usually by counting the number of TCP SYN traffic coming from each IP address in a particular span of time and flagging the traffic when a threshold is exceeded.  By using a slow DOS tool that sends requests via SSL, IDS has no idea that you’re sending it slow DOS traffic.
• Stay out of the “Crowbar Hotel”:  Use the Ion Cannon, make logs on the target system, go to jail.  Use slow DOS with TOR and SSL, leave less traces, avoid having friends that will trade you for a pack of cigarettes.

Defenses

This part is fun, and by that I mean “it sucks”.  There are some things that help, but there isn’t a single solution that makes the problem go away.

• Know how to detect it.  This is the hard one.  What you’re looking for is Apache spawned out to MaxClients but not logging a comparative volume of traffic.  IE, the servers are hung up waiting for that one last request to finish and shucking all other requests.
• “ps aux | grep apache2 | grep start | wc -l” is equal to MaxClients +2.
• Your webserver isn’t logging the normal amount of requests.  Use some grep-foo and “wc -l” to compare traffic from: a month ago, a day ago, an hour ago, and the last 5 minutes.
• Disable POST as a method if you don’t need it.  Some of the more advanced techniques rely on the fact that POST can contain more headers and more body data.
• Use an astronomically high number of servers.  If your server processes can timeout and respawn faster than the slow DOS can hang them, you win.  If you had maybe 3000 servers, you wouldn’t have to worry about this.  Don’t have 3000 servers, I might have some you could use.
• Set a lower connection timeout.  Something like 15-30 seconds will keep Apache humming along.
• Limit the request size.  1500 bytes is pretty small, 3K is a pretty good value to set.  Note that this needs testing, it will break some things.
• Block TOR exit nodes before the traffic reaches your webservers (IE, at layer 3/4).  TOR has a list of these.

• Apache Killer Effects on Target
• Stress-Test Apache with Intent to Tune: BSOFH Tip for the Software Masochist
• Keeping Up With the DDoS Kids
• Training the Apache Killers
• A Little Story About a Tool Named #RefRef

So since I’ve semi-officially been granted the title of “The DDoS Kid” after some of the incident response, analysis, and talks that I’ve done, I’m starting to get asked a lot about how much the average DDoS costs the targeted organization.  I have some ideas on this, but the simplest way is to recycle Business Continuity/Disaster Recovery figures but with some small twists.

Scoping:

• Plan on a 4-day attack.  A typical attack duration is 2-7 days.
• Consider an attack on the “main” (www) site and anything else that makes money (shopping cart, product pages)

Direct:

• Downtime: one day’s worth of downtime for both peak times (for most eCommerce sites, that’s Thanksgiving to January 5th) and low-traffic times x  (attack duration).
• Bandwidth: For services that charge by the bit or CPU cycle such as cloud computing or some ISP services, the direct cost of the usage bursting.  The cost per bit/cpu/$foo is available from the service provider, multiply your average rate for peak times by 1000 (small attack) or 10000 (large attack) x (attack duration) worth of usage.  This is the only big difference in cost from BCP/DR data.
• Mitigation Services:  Figure $5K to $10K for a DDoS mitigation service x (duration of attack).

Indirect:

• Increased callcenter load: A percentage (10% as a starting guess) of user calls to the callcenter x (average dollar cost per call) x (attack duration).
• Increased physical “storefront” visits: A percentage (10%) of users now have to go to a physical location x (attack duration).
• Customer churn: customer loss due to frustration.  Figure 2-4% customer loss x (attack duration).

Brand damage, these vary from industry to industry and attack to attack:

• Increased marketing budget: Percentage increase in marketing budget.  Possible starting value is 5%.
• Increased customer retention costs: Percentage increase in customer retention costs.  Possible starting value is 10%.

Note that it’s reasonably easy to create example costs for small, medium, and large attacks and do planning around a medium-sized attack.

However we recycle BCP/DR figures for an outage, mitigation of the attack is different:

• For high-volume attacks, you will need to rely on service providers for mitigation simply because of their capacity.
• Fail-over to a secondary site means that you now have two sites that are overwhelmed.
• Restoration of service after the attack is more like recovering from a hacking attack than resuming service at the primary datacenter.

• DDoS and Elections
• Keeping Up With the DDoS Kids
• DojoCon DDoS Video
• The Rise of the Slow Denial of Service
• A Little Story About a Tool Named #RefRef

My DDoS presentation at DojoCon on Sunday.  A big thanks to Marcus J Carey for organizing the con and Adrian Crenshaw for doing the recording.

• DojoCon 2009 Presentation
• Barcode Hacking
• DDoS and Elections
• Massively Scaled Security Solutions for Massively Scaled IT
• DDoS Planning: Business Continuity with a Twist

Inspired by Anonymous, Operation Payback, and the “DDoS attacks as a legitimate form of protest?” article at ZDNet

• Conflicker ala IKANHAZFIZMA
• Noms and IKANHAZFIZMA
• IKANHAZFIZMA Finds Caution Tape
• Training the Apache Killers
• #RefRef the Vaporware DoS Tool

Nope, we’re not going to talk about ego trips, hidden agendas, or complete irresponsible transparency.  This blog post is about some of the fallout inside the Government security teams.

The powers that be would like to remind you that downloading classified documents off the Intertubez does not make them unclassified.  An anonymous source that I talked to last week gave me the info that they were busy tracking their users’ browsing behaviors so that if you (the hypothetical you) went to WikiLeaks and downloaded a classified document, the InfoSec goon squad would show up outside your cubicle to shred your hard drive because you had just been responsible for a classified spillage–ie, your unclassified desktop now has classified material on it and as per procedure the only way to deal with the situation is to overwrite your hard drive and reimage it.  I have a couple thoughts about this:

• Where were the InfoSec goons when their users were getting drive-by malware from questionable sites?
• If it’s on TV, it’s not a “secret” anymore.
• Don’t our InfoSec teams have something better they can spend their time doing other than being the WikiLeaks monitor?

And then there’s the Ambulance Chasing Department.  According to a different anonymous source, the vendors have descended upon the State Department hawking their security solutions, including this gem of a webinar.  Not quite sure what the webinar is on, except that they’re targeting you to sell something.

From: Prism Microsystems

Sent: Wednesday, December 01, 2010 10:01 AM

To: user@state.gov

Subject: Webinar: Prevent “WikiLeaks-type” Data Loss

Webinar:  How to Prevent “WikiLeaks-type” Data Loss in Government Networks

Following the most recent publication of classified documents by WikiLeaks, government agencies are reviewing current provisions for protecting classified and top secret data – they are also researching best practices and alternative methods to monitor, prevent, and document data loss.

Attend this webinar to learn:

• how the leaks happened
• telltale signs of a leak
• what you can do to prevent them

Leak picture by jillallyn.

• Transparency in Government: Just Give us the Data!
• Workin’ for the ‘Counters: an Analysis of my Love-Hate Relationship with the CPAs
• When Standards Aren’t Good Enough
• Preliminary Findings on Cybersecurity Review Now Out
• Federated Vulnerability Management

So last week Anonymous staged a Distributed Denial of Service against the RIAA, MPAA, and Aiplex over Aiplex’s announcement that they were launching DDoS attacks of their own against torrent sites sharing copyrighted content.  In doing a bit of research on techniques that were being used, I came across this wonderful bit of script:

while true; do wget ‘http://riaa.com/goldandplatinumdata.php?table=SEARCH_RESULTS&title=&artist=&label=&format=&category=&type=&awardDescription=&startMonth=1&startYear=0&endMonth=12&endYear=2009&sort=Date&sense=ASC&perPage=5000000000&go=Search’ > /dev/null &  done

For those of you who don’t get it just yet, this is an awesome attack to study.  Basically what it does is to run an endless loop grabbing a search page url with parameters that will do a huge database query.  The beauty of the attack is that it’s “highly leveraged”: for one http get, you generate a sizeable amount of database load.  Now multiply that by thousands of yahoos out there running the same script, and it will rollover the target’s database server and possibly the applications servers that query it.

• A Little Story About a Tool Named #RefRef
• The Rise of the Slow Denial of Service
• Apache Killer Effects on Target
• DDoS and Elections
• DDoS Planning: Business Continuity with a Twist