I’m curtailing my blog for a couple of weeks.  I’m busy helping out with Haiti.

I spent last Saturday at CrisisCamp DC.  It’s a barcamp-style hackathon to build applications to help relief workers in Haiti.  Think long-range wifi routers to network the country where the infrastructure is destroyed.  Think a website for quake survivors to tell their story.  Think a Craiglist for relief workers where somebody with an oxygen generator and  somebody with a power supply can get together and make something that helps both of them.  Think all of these created in an 8-hour development stint.

Yes, security folks, you can help.  Not only that, but you have the technical skills to get web apps stuff done and the project management experience to lay out what it is that needs to be done.

We’re holding another CrisisCamp in DC this Saturday the 30th.

Go to crisiscommons.org and look for a project that interests you or a local camp.

Here, let Andy Carvin break it all down “Big Bird Style”:

Movie by @Digiphile, Alex Howard from SearchCompliance.com.  Hopefully I didn’t just “out” him.  =)

Bookmark to:

Hide Sites

So I’m working with the AppSecDC folks doing press relations amongst other things.  I’ve noticed several themes for the conference that might be of interest to the rest of the world.  Of course there’s the usual “The end is nigh, and not even Norton can save you!!!!!” stuff that’s been the staple of security conferences for the past 5 years or so (oh noes, teh Internetz are broken.  Again)

However, AppSecDC has another set of themes that are mostly unique to OWASP and AppSecDC in particular:

• The OWASP Approach to Security: it’s not process/product, it’s education and outreach.  Thanks to Doug Wilson for this idea.  Basically with host and network security, the option is to buy stuff and throw it at the problem.  With application security, it’s “go out and touch a developer today” and “use ESAPI as a tool to help the developers write better and secure code more quickly”.  This is a new concept to the system integrator that I am, but I like it much better than my usual approach.
• Government and Application Security: we’re about 5 years behind industry, how do we catch up?  To this effort, we have some notable Government speakers such as a keynote by Joe Jarzombek, Director for Software Assurance in the National Cyber Security Division of the U.S. Department of Homeland Security.
• OWASP Top 10 2009/2010: This will be announced at AppSecDC with much w00tness and excitement.
• OWASP National Summit: this will be held the day before the official conference.

Convinced you want to go?  Check out the conference site.

Bookmark to:

Hide Sites

My presentation slides from Sector 2009.  This was a really fun conference, the Ontario people are really, really nice.

Presentation Abstract:

The US Federal Government is the world’s largest consumer of IT products and, by extension, one of the largest consumers of IT security products and services. This talk covers some of the problems with security on such a massive scale; how and why some technical, operational, and managerial solutions are working or not working; and how these lessons can be applied to smaller-scale security environments.

Bookmark to:

Hide Sites

So I’ve been having some problems with my server for a month or so–periodically the number of apache servers would skyrocket and the box would get so overloaded (load of ~50 or so) that I couldn’t even run simple commands on it.  I would have to get into the hardware console and give the box a hard boot (a graceful reboot wouldn’t work).

Root cause is I’m a dork, but more about that later.

Anyway, I needed a way to troubleshoot and fix it.  The biggest problem I had was that the problem was very sporadic–sometime it would be 2 weeks between crashes, other times it would be 3 times in one day.  This is so begging for a stress-test really badly.  Looking on the Internet, I found a couple of articles about running a load-tester on apache and information on the tuning settings but not really much about a methodology (yeah yeah I work for a Big 4 firm, the word still makes me shudder even though it’s the right one to use here) to actually solve the problem of apache tuning.

So the “materials” I needed:

• One server running apache.  Mine runs Apache2 under Debian Stable.  This is a little bit different from the average distro out there in that the process is apache2 and the command is apache2ctl where normally you would have httpd and httpdctl.  If you try this at home, you’ll need to use the latter commands.
• An apache tuning guide or 3.  Here’s the simplest/most straightforward one I’ve seen.
• A stress-tester.  Siege is awesome for this.
• Some simple shell commands: htop (top works here too), ps, grep, and wc.

Now for the method to my madness…

I ssh into my server using three different sessions.  On one I run htop.  Htop is a version of top that gives you a colored output and supports multiple processors.  The output without stress-testing looks something like this:

(Click for a life-size image)

I keep one session free to edit files and do an emergency “killall apache2″ if things get out of hand (and they will really quickly, I had to pull the plug about 20 times throughout this process).   I run a simple command on another ssh session to get a count of how many apache threads I have running:

rybolov@server:~$ ps aux | grep apache2 | grep start | wc -l
11

OK, so far so good.  I’ve got 11 threads running with no load and RAM usage of 190MB.  I needed the extra “grep start” because it removes the text editor I have open on apache2.conf and anything else I might be doing in the background.

I also killed apache, waited 10 seconds, and looked at the typical RAM use.  With no apache running, I use about 80MB just for the OS and everything else I’m running.  This means that I’m using 110MB of RAM for 11 apache threads, which means I’m using ~10MB of RAM for each apache thread.  Now that’s something important I can use.

I took my tuning settings in apache2.conf (httpd.conf for most distros) (Apache2 uses the prefork module which uses threads, read the tuning guide for more info) and set them at the defaults listed in the tuning guide.  They became something like the following:

<IfModule prefork.c>
  StartServers            8
  MinSpareServers         5
  MaxSpareServers        20
  MaxClients            150
  MaxRequestsPerChild  1000
</IfModule>

Notice how the MaxClients is set at 150?  This will prove to be my downfall later.  Turns out that my server is RAM-poor for as much processor as it has or Wordpress is a RAM hog (or both, which is the case =)  ).  I’ll eventually upgrade my server, but since it’s a cloud server from Mosso, I pay by the RAM and drive space.

After each edit of apache2.conf, you need to give apache a configuration test and reload:

server:~# apache2ctl configtest
Syntax OK                        <- If something else comes back, fix it!!
server:~# apache2ctl restart

I’m now ready to stress-test using the default setup.  This is the awesome part.  First, I need to simulate a load.  I make an url seedfile so that siege will bounce around between a handful of pages.  I make a file siege.urls.txt and put in a collection of urls so that it looks like the following:

http://www.guerilla-ciso.com/

http://www.guerilla-ciso.com/about

http://www.guerilla-ciso.com/contact

http://www.guerilla-ciso.com/papers-and-presentations

....<about 20 lines deleted here, you get the point>

http://www.guerilla-ciso.com/page/2

http://www.guerilla-ciso.com/page/3

http://www.guerilla-ciso.com/page/4

I’m sure there is an efficient and fun way to make this, like say, a text-only sitemap or sproxy which is made by the same guy who does siege, but since I only needed about 30 urls, I just cut-n-pasted them off the blog homepage.

I fire up siege and give my webserver a thorough drubbing, running 50 connections for 10 minutes and using my url seedfile.  BTW, I’m running siege on the webserver itself, so there isn’t anything in the way of network latency.  <enter sinister laugh of evil as I sadistically torture my apache and the underlying OS>

server:~# siege -c 50 -t 600s -f siege.urls.txt
** SIEGE 2.66
** Preparing 50 concurrent users for battle.    <-The guy writing siege has a wicked sense of humor.
The server is now under siege...                <-Man the ramparts, Apache, they're coming for you!
HTTP/1.1 200   1.08 secs:   16416 bytes ==> /
HTTP/1.1 200   1.07 secs:   16416 bytes ==> /
....<about 2 bazillion lines deleted here, you get the idea>
HTTP/1.1 200   4.66 secs:    8748 bytes ==> /about
HTTP/1.1 200   3.92 secs:    8748 bytes ==> /about
Lifting the server siege...      done.

Transactions:                  61 hits   <-No, this isn't actual, I abbreviated the siege output
Availability:              100.00 %      <-with a ctrl-c just to get some results so I didn't
Elapsed time:                6.70 secs   <-have to scroll through all that output from the real test.
Data transferred:            0.87 MB
Response time:                3.27 secs
Transaction rate:            9.10 trans/sec
Throughput:                0.13 MB/sec
Concurrency:               29.75
Successful transactions:          61
Failed transactions:               0
Longest transaction:            5.61
Shortest transaction:            1.07

Now I watch the output of htop.  Under stress, the output looks something like this:

(Click for a life-size image)

Hmm, looks like I have a ton of apache threads soaking up all my RAM.  What happens is that in about 30 seconds, the OS starts swapping and the swap use just keeps growing until the OS is unresponsive.  This is a very interesting cascade failure because writing to swap incurs a load which makes the OS write to swap more.  Maybe I need to limit either the amount of RAM used per apache or limit the maximum amount of threads that apache spawns.  The tuning guide tells us how…

There is one setting that is the most important in tuning apache, it’s MaxClients.  This is the maximum number of servers (with the worker module) or threads (prefork module).  Looking at my apache tuning guide, I get a wonderful formula: ($SizeOfTotalRAM – $SizeOfRAMForOS) / $RAMUsePerThread = MaxClients.  So in my case, (512 – 80) / 11 = 39.something.  Oops, this is a far cry from the 150 that comes as default.  I also know that the RAM/thread number I used was without any load on apache, so with a load on and generating dynamic content (aka Wordpress) , I’ll probably use ~15MB per thread.

One other trick that I can use:  Since I think that what’s killing me is the number of apache threads, I can run with a reduced amount of simultaneous connections and watch htop.  When htop shows that I’ve just started to write to swap, I can run my ps command to find out how many apache threads I have running.

rybolov@server:~$ ps aux | grep apache2 | grep start | wc -l
28

Now this is about what I expected:  With 28 threads going, I tipped over into using swap.  Reversing my tuning formula, I get (28 threads x 15 MB/thread) +80 MB for OS = 500 MB used.  Hmm, this makes much sense to me, since the OS starts swapping when you use ~480MB of RAM.

So I go back to my prefork module tuning.

<IfModule mpm_prefork_module>
 StartServers          8
 MinSpareServers       5
 MaxSpareServers      10
 MaxClients           25
 MaxRequestsPerChild   2000
</IfModule>

I set MaxClients at 25 because 28 seems to be the tipping point, so that gives me a little bit of “wiggle room” in case something else happens at the same time I’m serving under a huge load.  I also tweaked some of the other settings slightly.

Then it’s time for another siege torture session.  I run the same command as above and watch the htop output.  With the tuning settings I have now, the server dips into swap about 120MB and survives the full 10 minutes.  I’m sure the performance is degraded somewhat by going into swap, but I’m happy with it for now because the server stays alive.  It wasn’t all that smooth, I had to do a little bit of trial and error first, starting with MaxClients 25 and working my way up to 35 under a reduce siege load (-c 25 -t 60s) to see what would happen, then increasing the load from siege (-c50 -t 600s) and ratcheting MaxClients back down to 25.

And as far as me being a dork… well, aside from the huge MaxClients setting (That’s the default, don’t blame me), I set MaxRequestsPerChild to 100 instead of 1000, meaning that every 100 http requests I was rolling over and making a new thread.  That would lead to cascade failure under a load. (duh!)

Bookmark to:

Hide Sites

Rybolov’s Note: Hello all, I’m venturing into an open-ended series of blog posts aimed at starting conversation. Note that I’m not selling anything *yet* but ideas and maybe some points for discussion.

Let’s get this out there from the very beginning: I agree with Ranum that full-scale, nation-v/s-nation Cyberwar is not a reality.  Not yet anyway, and hopefully it never will be.  However, on a smaller scale with well-defined objectives, cyberwar is not only happening now, but it is also a natural progression over the past century.

Looking at where we’re coming from in the existing models and techniques for activities similar to cyberwar, it frames our present state very nicely :

Electronic Countermeasures. This has been happening for some time.  The first recorded use of electronic countermeasures (ECM) was in 1905 when the Russians tried to jam radio signals of the Japananese fleet besieging Port Arthur.  If you think about ECM as DOS based on radio, sonar, etc, then it seems like cyberwar is just an extension of the same denial of communications that we’ve been doing since communication was “invented”.

Modern Tactical Collection and Jamming. This is where Ranum’s point about spies and soldiers falls apart, mostly because we don’t have clandestine operators doing electronic collection at the tactical level–they’re doing both collection and “attack”.  The typical battle flow goes something along the lines of scanning for items of interest, collecting on a specific target, then jamming once hostilities have begun.  Doctrinally, collection is called Electronic Support and jamming is called Electronic Attack.  What you can expect in a cyberwar is a period of reconnaissance and surveillance for an extended length of time followed by “direct action” during other “kinetic” hostilities.

Radio Station Jamming. This is a wonderful little world that most of you never knew existed.  The Warsaw Pact used to jam Radio America and other sorts of fun propaganda that we would send at them.  Apparently we’ve had some interesting radio jamming since the end of the Cold War, with China, Cuba, North Korea, and South Korea implicated in some degree or another.

Website Denial-of-Service. Since only old people listen to radio anymore and most news is on the Internet, so it makes sense to DOS news sites with an opposing viewpoint.  This happens all the time, with attacks ranging from script kiddies doing ping floods to massive DOSBots and some kind of racketeering action… “You got a nice website, it would be pretty bad if nobody could see it.”  Makes me wonder why the US hasn’t taken Al Jazeera off the Internet.  Oh, that’s right, somebody already tried it.  However, in my mind, jamming something like Al Jazeera is very comparable to jamming Voice of America.

Estonia and Gruzija DOS. These worked pretty well from a denial-of-communications standpoint, but only because of the size of the target.  And so what if it did block the Internet, when it comes to military forces, it’s at best an annoyance, at most it will slow you down just enough.  Going back to radio jamming, blocking out a signal only works when you have more network to throw at the target than the target has network to communicate with the other end.  Believe it or not, there are calculators to determine this.

Given this evolution of communications denial, it’s not unthinkable that people wouldn’t be launching electronic attacks at each other via radar, radio, carrier pigeon, IP or any other way they can.

However, as in the previous precedents and more to some of the points of Ranum’s talk at DojoSec, electronic attacks by themselves only achieve limited objectives.  Typically the most likely type of attack is to conduct a physical attack and use the electronic attack, whether it’s radio, radar, or IT assets, to delay the enemy’s response.  This is why you have to take an electronic attack seriously if it’s being launched by a country which has a military capable of attacking you physically–it might be just a jamming attack, it might be a precursor to an invasion.

Bottom line here is this: if you use it for communication, it’s a target and has been for some time.

Bookmark to:

Hide Sites

So how does the Guerilla-CISO staff communicate with the locals on jaunts to foreign lands such as Deleware, New Jersey, and Afghanistan?  The answer is simple, we use interpreters, known in infantrese as “terps”.  Yes, you might not trust them deep down inside because they harbor all kinds of loyalties so complex that you can spend the rest of your life figuring out, but you can’t do the job without them.

But in remembering how we used our interpreters, I’m reminded of some basic concepts that might be transferable to the IT security and risk management world.  Or maybe not, at least kick back and enjoy the storytelling while it’s free. =)

Know When to Treat Them Like Mushrooms: And by that, we mean “keep them in the dark and feed them bullsh*t”.  What really mean is to tell potentially adversarial people that you’re working with the least amount of information that they need to do their job in order to limit the frequency and impact of them doing something nasty.  When you’re planning a patrol, the worst way to ruin your week is to tell the terps when you’re leaving and where you’re going.  That way, they can call their Taliban friends when you’re not looking and they’ll have a surprise waiting for you.  No, it won’t be a birthday cake.  The way I would get a terp is that one would be assigned to me by our battalion staff and the night before the patrol I would tell the specific terp that we were leaving in the morning, give them a time that I would come by to check up on them, and that they would need to bring enough gear for 5 days.  Before they got into my vehicles and we rolled away, I would look through their gear to make sure they didn’t have any kind of communications device (radio or telephone) to let their buddies know where we were at.

Fudge the Schedule to Minimize Project Risk: Terps–even the good ones–are notorious for being on “local time”, which for a patrol means one hour later than you told them you were leaving.  The good part about this is that it’s way better than true local time, which has a margin of error of a week and a half.  In order to keep from being late, always tell the terps when you’ll need them an hour and a half before you really do, then check up on them every half hour or so.  Out on patrol, I would cut that margin down to half an hour because they didn’t have all the typical distractions to make them late.

Talk Slowly, Avoid Complex Sentences: The first skill to learn when using terps is to say things that their understanding of English can handle.  When they’re doing their job for you, simple sentences works best.  I know I’m walking down the road of heresy, but this is where quantitative risk assessment done poorly doesn’t work for me because now I something that’s entirely too complex to interpret to the non-IT crowd.  In fact, it probably is worse than no risk assessment at all because it comes accross as “consultantspeak” with no tangible link back to reality.

Put Your Resources Where the Greatest Risk Is: To a vehicle patrol out in the desert, most of the action happens at the front of the patrol.  That’s where you need a terp.  That way, the small stuff, such as asking a local farmer to move his goats and sheep out of the road so you can drive through, stays small–without a terp up front, a 2-minute conversation becomes 15 minutes of hassle as you first have to get the terp up to the front of the patrol then tell them what’s going on.

Pigs, Chicken, and Roadside Bombs: We all know the story about how in the eggs and bacon breakfast, the chicken is a participant but the pig is committed.  Well, when I go on a patrol with a terp, I want them to be committed.  That means riding in the front vehicle with me.  It’s my “poison pill” defense in knowing that if my terp tipped off the Taliban and they blow up the lead vehicle with me in it, at least they would also get the terp.  A little bit of risk-sharing in a venture goes a long way at getting honesty out of people.

Share Risk in a Culturally-Acceptable Way: Our terps would balk at the idea of riding in the front vehicle most of the time.  I don’t blame them, it’s the vehicle most likely to be turned into 2 tons of slag metal thanks to pressure plates hooked up to IEDs.  The typical American response is something along the lines of “It’s your country, you’re riding up front with me so if I get blown up, you do to”.  Yes, I share that ideal, but the Afghanis don’t understand country loyalties, the only thing they understand is their tribe, their village, and their family.  The Guerilla-CISO method here is to get down inside their heads by saying “Come ride with me, if we die, we die together like brothers”.  You’re saying the same thing basically but you’re framing it in a cultural context that they can’t say no to.

Reward People Willing to Embrace Your Risks: One of the ways that I was effective in dealing with the terps was that I would check in occassionally to see if they were doing alright during down-time from missions.  They would show me some Bollywood movies dubbed into Pashto, I would give them fatty American foods (Little Debbie FTW!).  They would play their music.  I would make fun of their music and amaze them because they never figured out how I knew that the song had drums, a stringed instrument, and somebody singing (hey, all their favorite songs have that).  They would share their “foot bread” (the bread is stamped flat by people walking on it before it’s cooked, I was too scared to ask if they washed their feet first) with me.  I would teach them how to say “Barbara (their assignment scheduler back on an airbase) was a <censored> for putting them out in the middle of nowhere on this assignment” and other savory phrases.  These forays weren’t for my own enjoyment, but to build rapport with the terps so that they would understand when I would give them some risk management love, Guerilla-CISO style.

Police, Afghan Army and an Interpreter photo by ME!.  The guy in the baseball cap and glasses is one of the best terps I ever worked with.

Bookmark to:

Hide Sites