I’ve noticed a trend over the past 6 months: DDoS traffic associated with elections.  A quick sampling of news will show the following:

• http://www.opendemocracy.net/od-russia/irina-borogan-andrei-soldatov/kremlin-and-hackers-partners-in-crime <-Russia
• http://www.theregister.co.uk/2012/03/26/hong_kong_vote_hack/ <-Hong Kong
• http://www.theregister.co.uk/2011/12/07/korean_election_ddos_row/ <-Korea
• http://www.cbc.ca/news/politics/story/2012/03/27/pol-ndp-voting-disruption-deliberate.html <-Canada (rybolov: yikes!)
• http://www.csoonline.com/article/700523/ddos-attackers-target-russian-election-webcams <-Russia again

Last week it picked up again with the re-inauguration of Vladimir Putin.

• http://en.ria.ru/russia/20120506/173265737.html
• http://www.washingtonpost.com/world/europe/peaceful-protest-turns-violent-in-moscow/2012/05/06/gIQAFti35T_story.html
• http://english.ruvr.ru/2012_05_06/73954450/
• http://www.rferl.org/content/independent_russian_daily_hit_by_dos_attack_kommersant/24571463.html

And then yesterday, Ustream and their awesome response: which, in the Rybolov-paraphrased version read something like: “We shall go on to the end. We shall fight in France, we shall fight on the Interblagosphere, we shall fight with growing confidence and growing strength in our blocking capabilities, we shall defend our videostreams, whatever the cost may be. We shall fight on the routers, we shall fight on the load balancers, we shall fight in the applications and in the databases, we shall fight by building our own Russian subsite; we shall never surrender!!!!1111″ (Ref)

Afghanistan Presidential Elections 2004 photo by rybolov.

So why all this political activity?  A couple of reasons that I can point to:

• Elections are a point-in-time.  It’s critical for one day.  Anything that has a short window of time is a good DDoS target.
• DDoS is easy to do.  Especially for the Russians.  Some of them already have big botnets they’re using for other things.
• Other DDoS campaigns.  Chaotic Actors (Anonymous and their offshoots and factions) have demonstrated that DDoS has at a minimum PR value and at the maximum financial and political value.
• Campaign sites are usually put up very quickly.  They don’t have much supporting infrastructure and full/paid/professional staffing.
• Elections are IRL Flash Mobs.  Traffic to a campaign site increases slowly at first then exponentially the closer you get to the day of the election.  This stresses what infrastructure is in place and design ideas that seemed good at the time but that don’t scale with the increased load.

So is this the future of political campaigns?  I definitely think it is.  Just like any other type of web traffic, as soon as somebody figures out how to use the technology for their benefit (information sharing => eCommerce => online banking => political fundraising), a generation later somebody else figures out how to deny that benefit.

How to combat election DDoS:

• Have a plan.  You know that the site is going to get flooded the week of the election.  Prepare accordingly.  *ahem* Expect them.
• Tune applications and do caching at the database, application, webserver, load balancer, content delivery network, etc.
• Throw out the dynamic site.  On election day, people just want to know a handful of things.  Put those on a static version of the site and switch to that.  Even if you have to publish by hand every 30 minutes, it’s better than taking a huge outage.
• Manage the non-web traffic.  SYN and UDP floods have been around for years and years and still work in some cases.  For these attacks, you need lots of bandwidth and something that does blocking: these point to a service provider that offers DDoS protection.

It’s going to be an interesting November.

• Keeping Up With the DDoS Kids
• DDoS Planning: Business Continuity with a Twist
• DojoCon DDoS Video
• The Rise of the Slow Denial of Service
• Apache Killer Effects on Target

Usually when you think about Denial of Service attacks nowadays, most people think up images of the Anonymous kids running their copy of LOIC in a hivemind or Russian Gangsters building a botnet to run an online protection racket.  Now there is a new-ish type of attack technique floating around which I believe will become more important over the next year or two: the slow http attacks.

Refs:

• Slowloris
• Slow HTTP POST
• RU Dead Yet (RUDY)
• Tor’s Hammer
• XerXes

How Slow DOS Works

Webservers run an interesting version of process management.  When you start an Apache server, it starts a master process that spawns a number of listener processes (or threads) as defined by StartServers (5-10 is a good starting number).  Each listener serves a number of requests, defined by MaxRequestsPerChild (1000 is a good number here), and then dies to be replaced by another process/thread by the master server.  This is done so that if there are any applications that leak memory, they won’t hang.  As more requests are received, more processes/threads are spawned up to the MaxClients setting.  MaxClients is designed to throttle the number of processes so that Apache doesn’t forkbomb and the OS become unmanageable because it’s thrashing to swap.  There are also some rules for weaning off idle processes but those are immaterial to what we’re trying to do today.

Go read my previous post on Apache tuning and stress testing for the background on server pool management.

What happens in a slow DOS is that the attack tools sends an HTTP request that never finishes.  As a result, each listener process never finishes its quota of MaxRequestsPerChild so that it can die.  By sending a small amount of never-complete requests, Apache gladly spawns new processes/threads up to MaxClients at which point it fails to answer requests and the site is DOS’ed.  The higher the rate of listener process turnover, the faster the server stops answering requests.  For a poorly tuned webserver configuration with MaxClients set too high, the server starts thrashing to swap before it hits MaxClients and to top it off, the server is unresponsive even to ssh connections and needs a hard boot.

The beauty of this is that the theoretical minimum number of requests to make a server hang for a well-tuned Apache is equal to MaxClients.  This attack can also take out web boundary devices: reverse proxies, Web Application Firewalls, Load Balancers, Content Switches, and anything else that receives HTTP(S).

Post photo by Salim Virji.

Advantages to Slow DOS Attacks

There are a couple of reasons why slow DOS tools are getting research and development this year and I see them growing in popularity.

• Speed and Simplicity:  Slow DOS attacks are quick to take down a server.  One attacker can take down a website without trying to build a botnet or cooordinate attack times and targets with 3000 college students and young professionals.
• TOR:  With volume-based attacks like the Low Orbit Ion Cannon, it doesn’t make sense to route attack traffic through TOR.  TOR adds latency, throttles the amount of requests that the attacker can send, and might eventually fail before the target’s network does.  Using TOR keeps the defender from tracking you back to your real location.
• Server Logging: Because the request is never completed, most servers don’t make a log.  This makes it very hard to detect or troubleshoot which means it takes longer to mitigate.  I’m interested in exceptions if you know specifics on which webserver/tool combinations make webtraffic logs.
• IDS Evasion: Most DOS tools are volume-based attack.  There are IDS rules to detect these: usually by counting the number of TCP SYN traffic coming from each IP address in a particular span of time and flagging the traffic when a threshold is exceeded.  By using a slow DOS tool that sends requests via SSL, IDS has no idea that you’re sending it slow DOS traffic.
• Stay out of the “Crowbar Hotel”:  Use the Ion Cannon, make logs on the target system, go to jail.  Use slow DOS with TOR and SSL, leave less traces, avoid having friends that will trade you for a pack of cigarettes.

Defenses

This part is fun, and by that I mean “it sucks”.  There are some things that help, but there isn’t a single solution that makes the problem go away.

• Know how to detect it.  This is the hard one.  What you’re looking for is Apache spawned out to MaxClients but not logging a comparative volume of traffic.  IE, the servers are hung up waiting for that one last request to finish and shucking all other requests.
• “ps aux | grep apache2 | grep start | wc -l” is equal to MaxClients +2.
• Your webserver isn’t logging the normal amount of requests.  Use some grep-foo and “wc -l” to compare traffic from: a month ago, a day ago, an hour ago, and the last 5 minutes.
• Disable POST as a method if you don’t need it.  Some of the more advanced techniques rely on the fact that POST can contain more headers and more body data.
• Use an astronomically high number of servers.  If your server processes can timeout and respawn faster than the slow DOS can hang them, you win.  If you had maybe 3000 servers, you wouldn’t have to worry about this.  Don’t have 3000 servers, I might have some you could use.
• Set a lower connection timeout.  Something like 15-30 seconds will keep Apache humming along.
• Limit the request size.  1500 bytes is pretty small, 3K is a pretty good value to set.  Note that this needs testing, it will break some things.
• Block TOR exit nodes before the traffic reaches your webservers (IE, at layer 3/4).  TOR has a list of these.

• Apache Killer Effects on Target
• Stress-Test Apache with Intent to Tune: BSOFH Tip for the Software Masochist
• Keeping Up With the DDoS Kids
• Training the Apache Killers
• A Little Story About a Tool Named #RefRef

My DDoS presentation at DojoCon on Sunday.  A big thanks to Marcus J Carey for organizing the con and Adrian Crenshaw for doing the recording.

• DojoCon 2009 Presentation
• Barcode Hacking
• DDoS and Elections
• Massively Scaled Security Solutions for Massively Scaled IT
• DDoS Planning: Business Continuity with a Twist

So for those of you keeping track at home:

• Indian firm Aiplex announced that they were launching Distribute Denial-of-Service (DDoS) attacks against The Pirate Bay. (the attack)
• The collective wisdom (Anonymous) of the Internet decided that in turn it would DDoS Aiplex, the MPAA, the RIAA, and their international counterparts in Operation Payback. (the counter-attack)
• Somebody has DDoS’ed the sites coordinating the attacks. (the counter-counter-attack)
• I’m popping up some popcorn to wait for the counter-counter-counter-attack and to watch the backscatter.

May we all live in interesting times, to say the least.  Some random thoughts I’m having about the DDoS campaigns:

• If people hate you enough to show up with signs outside your office to protest, they hate you enough to flood your network.
• Activist/vigilante/mob rule/protest has evolved to a very viable DDoS platform using a wide variety of operating systems.
• The DDoS campaign in 2008 against the Church of Scientology was called off by activist leaders, so now we’re seeing the unbridled fury of the Intertubez unleashed.
• On the tools side of things, I’ve seen some good development and some really creative methods to let non-technical folks to participate in the DDoS.
• Coordinating an activist army seems like the weak point in the model.

• Keeping Up With the DDoS Kids
• DDoS and Elections
• DojoCon DDoS Video
• The Rise of the Slow Denial of Service
• Apache Killer Effects on Target

So let me give you a hypothetical job:

• You have to give up your high-paying private-sector job to be a Government employee
• You have tons of responsibility
• You have no real authority
• You have no dedicated budget
• You have no staffers
• The job has had half a dozen people filling it in the last 7 years
• The job has been open longer than it’s been staffed over the past 7 years

And yet this is what we’re asking candidates to do in order to even be a candidate for the Cybersecurity Coordinator.  Yes, this is the exact same problem that all CISOs have with having a huge helping of responsibility and none of the authority to get things done, only we scaled it up and out to a national-level CISO position.

Somebody’s even gone as far to say that the lack of candidates for the job is the security field’s way of sending the message that you didn’t scope the job right.  I think this opinion has much merit.  CISOs being what they are, they’re usually pretty astute at walking into an ambush, and this job has all the makings of a good one.

I’ll even turn it around the other way and say that the security industry has yet to produce a CISO’s CISO–somebody who can do politics, budget, security, IT, and consensus-building all in one person.  We have lots of people who can manage the enterprise and below, but it’s that additional little bit of political intrigue that is what we’re missing.  Security people usually avoid politics like the bubonic plague because we’re an industry full of people who say it like it really is.  This is a detriment in sales and politics.

So in true Guerilla-CISO fashion of not pointing out problems without offering something as a fix (no matter how much of a strawman arguement it really is), this is what we need to do to get people interested in being the Cybersecurity Czar^wCoordinator:

• A really well-defined scope.  One person cannot do everything that we are asking for at this price (or any price for that matter).
• A budget for an operating staff where the number is more than than 8 digits.
• Statutory authority over the various departments and agencies responsible for cybersecurity: NCSD, S&T, DoJ, FBI, Commerce.  Indirect influence doesn’t work here, never has.
• The direct ear of the President.  Councils are OK, but puhlease, you want to get the job done, this is what it will take.

Then I read back through my list and realized that we really do need a law to create the Cybersecurity Czar position with everything that I just mentioned.  But here’s the rub: legislation is slow, the bills to make the Cybersecurity Czar aren’t even going to be looked at until the next congressional session because we’re still trying to figure out the budget for last year.

I also think that what we’re calling the Cybersecurity Czar is really 2 jobs.  You need somebody working for the Government CIO Vivek Kundra as the executive-branch CISO and you need a more senior person who worries about the military-industrial base, the critical infrastructure, the support to American commerce, and the protection of little old grandmas who represent the end-users.

Tsar’s Cannon photo by Siyad Ma.  Now that’s some teeth for the position.

• Blow-By-Blow on S.773–The Cybersecurity Act of 2009–Part 5
• Blow-By-Blow on S.773–The Cybersecurity Act of 2009–Part 4
• Next Up in Security Legislation: S3474
• Cyber Security coming to a boil
• A Layered Model for Massively-Scaled Security Management

In the military, there is a saying: “You go to war with the army you have, not with the army you wish you had.”  In other words, you do all your training in peace and once you go off to war, it’s too late to fix it. Not that I agree with all the Cyber Pearl Harbor doomsayers, but I think that the CyberArmy we got now isn’t the right one for the job.

So, let’s talk about services firms, contractors fit into this nicely since, well, they perform services.

There are 4 types of work that services firms do (and contractors are services firms):

• Brains: nobody else has done this before, but we hire a whole bunch of PhD people who can research how to get this done.  We charge really high prices but it’s because in the downtime, our people are doing presentations, going to symposiums, and working on things that you don’t even know exist.  Think old-school L0pht.  Think half of Mitre.  Think sharks with friggin laser beams, lasing and eating everything in sight.
• Gray Hair: We’ve done this before and know most of the problems that we can experience, along with the battle scars to prove it.  We charge quite a bit because we’re good and it takes less of us to get it done than our competitors.  Think most good IT engineers.  Think DLP and DAM right now.  Think infantry platoon sergeants.
• Procedural: There is a fairly sizeable market starting to grow around this service so we have to standardize quite a bit to reduce our costs to provide the service.  We use methodologies and tools so that we can take an army of trained college graduates, put them in a project, and they can execute according to plan.  Think audit staff.  Think help desk staff.  Think of an efficient DMV.
• Commodity: There isn’t a differentiator between competitors, so companies compete on price.  The way you make money is by making your cost of production lower or selling in volume.  Think Anti-Virus software (sorry friends, it’s true).  Think security guards.  Think peanut butter.

This is also the maturity model for technology, so you can take any kind of tech, drop it in at the top, and it percolates down to the bottom.  Think Internet use: First it was the academics, then the contractors, then the technology early adopters on CompuServe, then free Internet access to all.  For most technology, it’s a 5-10 year cycle to get from the top to the bottom.  You already know this: the skills you have now will be obsolete in 5 years.

Procedural Permit Required photo by Dawn Endico.

Now looking at government contracting….

As a government contractor, you are audited financially by DCAA and they add up all your costs and let you keep a fixed margin of around 13-20%.  You can pull some Stupid Contractor Tricks ™ like paying salaries and working your people 60 hours/week (this is called uncompensated overtime), but there still is a limit to what you can do.

This fixed margin forces you into high-volume work to turn a profit.  This in turn forces you into procedural or even commodity work.

If your project is strictly time and material, you make more money off the cheaper folks but for quality of work reasons, you have to provide them with a playbook of some sort.  This pushes you directly into the procedural tier.

There are some contractors providing services at the Brains and Gray Hair stages, only they are few and far between.

Traditional types of contractor security services:

• Security Program Management and Governance
• Audit and Penetration Testing
• Compliance and Certification and Accreditation Support
• Security Operations (think Managed Security Services)

Then back around to cyberwar…

Cyberwar right now is definitely at the top of the skill hierarchy.  We don’t have an official national strategy.  We have a Cybersecurity Coordinator that hasn’t been filled yet.  We need Brains people and their skills to figure this out.  In fact, we have a leadership drought.

And yet the existing contractor skillset is based on procedural offerings.  To be honest, I see lots of people with cybersecurity offerings, but what they really have is rebranded service offerings because the skills sets of the workforce haven’t changed.

Some of the procedural offerings work, but only if you keep them in limited scope.  The security operations folks have quite a few tranferable skills, so do the pen-testers.  However, these are all at the tactical level.  The managerial skills don’t transfer really at all unless you have people that are just well-rounded, usually with some kind of IT ops background.

But, and this is the important thing, we’re not ready to hire contractors until we do get some leadership in place. And that’s why the $25M question right now is “Who will that person be?”  Until that time, anything from the vendors and contractors is just posturing.

Once we get a national leadership and direction, then it’s a matter of lining up the services being offered with the needs at the time.  What I think we’ll find out at that time is that we’re grossly underrepresented in some areas and sadly underrepresented in some areas and that these areas are directly inverse to the skills that our current workforce has.  This part scares me.

We need workforce development.  There are some problems with this, mostly because it takes so long to “grow” somebody with the skills to get the job done–maybe 5-10 years with education and experience.  Sadly, about the time we build this workforce, the problem will have slid down the scale so that procedural offerings will probably work.  This frustrates me greatly.

The summary part…

Well, just like I don’t want to belong to any club that would stoop so low to have me as a member, it could be possible that almost all the contractors offering services aren’t the people that you want to hire for the job.

But then again, we need to figure out the leadership part first.  Sadly, that’s where we need the most love.  It’s been how many months with a significant leadership vacuum?  9? 12? 7 years?

The most critical step in building a cyberwar/cyberdefense/cyberfoo capability is in building a workforce.  We’re still stuck with the “option” of building the airplane while it’s taxiing down the runway.

• Security Assessment Economics
• Engagement Economics and Security Assessments
• A Little Advice From Mike and Lee
• Inside the Obama Administration’s Cyber Security Agenda
• How to Not Let FISMA Become a Paperwork Exercise