If you're new here and would like to see more of what I'm saying, you may want to subscribe to my RSS feed (I can even email my blog posts to you when I publish a new one) or have a look at my papers and presentations page for downloads of stuff that you can share or "borrow heavily from". You also might find my guidelines for posting comments interesting, especially if you're a government employee. If you want to see me blog about anything in particular, drop me a private email on how you think I'm completely full of myself, extend me an invitation to speak at your next security meeting/event, or just to ship a huge bag of money in my direction, you can do that through my contact page. Thanks for visiting and happy hacking!

In part 1 on this blog I outlined the fact penetration testing evolved from a grey-art practiced by hackers into a more formal process.  This evolution has created a bifurcation within “boutique” penetration test service providers.

On the one hand tools-oriented budget firms offer little value added beyond simply running simple vulnerability scans.  On the other more profession and experienced firms offer the same tests and scans but also offer analysis that can be offered as direct actionable input into an organization’s existing security governance structure. 

The fly in the ointment is that not all security consumers or security organizations are created equally.  Some IT security organizations can be characterized a compliance-based.  That is to say that they establish and follow a set of rule that they believe will put them on the road to IT security.

On the other hand, most IT security organizations are risk-based and technically oriented.  They also follow a formal structure but, addressing risk with the appropriate application of process, procedures, and technology.  In  graphical terms the situation would appear to line-up as depicted in table 1.  Table quadrant 1 representing a weak security organization supported by, “Tool-boys” is noted in red because the risks associated with this coupling.  Quadrants 2 and 3 are noted in yellow because of the risks associated with either a weak security organization or weak testing input.  

Table 1

However, in the real world the table should look more like Table 2. With the increasing acceptance of Compliance-based security models, a set of independently administered vulnerability scans suffices to “check the box” for the requirements for a penetration test.  This is good news for these budget “boutique” firms. 

Table 2

However, as might be expected, it is bad news for IT security in general since all networks live in the same security ecosystem.   Market drivers that encourage poor security practices hurt us all.

Hacker Store photo by LatinSuD.

Bookmark to:

Hide Sites

Penetration testing is a controversial topic with an interesting history. It is made all that much more controversial and perplexing because of an common disconnect between the service provider and the consumer.

Penetration started as a grey-art that was often practiced/delivered in an unstructured and undisciplined manner by reformed or semi-reformed hackers. Penetration testers used their own techniques and either their own home-grown tools or tools borrowed or traded with close associates. There was little reproducibility or consistency of results or reporting. As a result, the services were hard to integrate into a security program.

As the art evolved it became more structure and disciplined and tools, techniques, and reporting became more standardized. This evolution was driven by papers, articles, technical notes that were both formally published and informally distributed. In the end, a standardized methodology emerged that was largely based on the disciplined approach used by the most successful hackers.

Hakker Kitteh photo by blmurch.

At about the same time open-source, government and commercial tools began to emerge that automated many of the steps of the standardized methodology. These tools had two divergent impacts on the art of penetration testing. As these tools were refined and constantly improved they reinforced the standard methodology, provided more consistent and reproducible results and improved and standardized penetration reporting. All of this made penetration testing easier for the consumer to absorb and integrate into security programs. As a result, regulations and security protocols emerged that required penetration and security assessments. Nmap and Nessus are excellent examples of the kind of tools that help shape and push this evolution. And, because of their utility they are still indispensable tools today.

However, Nessus also helped to automate both data collection and analysis, it has lowered the bar for the skills and experience needed to conduct portions of the penetration testing methodology. This lowered the cost of penetration testing and made them much more broadly available. Thus, giving rise to so-called “boutique firms.” The problem with penetration testing “boutique firms” is that they fall into two broad categories; specialized highly professional firms led by experienced and technical security professionals who can translate automated tool output into root-cause analysis of vulnerabilities, and security program flaws. The second category of firm consists of opportunist firms with just enough knowledge to run automated tools and cut and paste the tool output into client reports. The later firms are some times called “tool-firms” and their employees “tool-boys.”

The later flourish for two reasons. The first is that they can offer their services at rock bottom prices. The second reason is that security organizations are often so ill-informed of the intricacies of the penetration testing process that can’t make a meaningful distinction between the professional firms and the tool-boys except on the basis of costs.

Bookmark to:

Hide Sites

I just got back from the SCAP 2008 conference at NIST HQ, and this is a collection of my thoughts in a somewhat random order:

Presention slides are available at the NVD website

I blogged about SCAP a year ago, and started pushing it in conversations with security managers that I came across.  Really, if you’re managing security of anything and you don’t know what SCAP is, you need to get smart on it really fast, if for no other reason than that you will be pitched it by vendors sporting new certifications.

Introduction to SCAP:  SCAP is a collection of XML schemas/standards that allow technical security information to be exchanged between tools.  It consists of the following standards:

• Common Platform Enumeration (CPE): A standard to describe a specific hardware, OS, and software configuration.  Asset information, it’s fairly humdrum, but it makes the rest of SCAP possible–think target enumeration and you’re pretty close.
• Common Vulnerabilities and Exposures (CVE): A definition of publicly-known vulnerabilities and weaknesses.  Should be familiar to most security researches and patch monkies.
• Common Configuration Enumeration (CCE): Basically, like CVE but specific to misconfigurations.
• Common Vulnerability Scoring System (CVSS): A standard for determining the characteristics and impact of security vulnerabilities.  Hmmm, sounds suspiciously like standardization of what is a high, medium, and low criticality vulnerability.
• Open Vulnerability and Assessment Language (OVAL):  Actually, 3 schemas to describe the inventory of a computer, the configuration on that computer, and a report of what vulnerabilites were found on that computer.
• Extensible Configuration Checklist Description Format (XCCDF): A data set that describes checks for vulnerabilities, benchmarks, or misconfigurations.  Sounds like the updates to your favorite vulnerability scanning tool because it is.

Hall of Standards inside NIST HQ photo by ME!!!

What’s the big deal with SCAP: SCAP allows data exchanges between tools.  So, for example, you can take a technical policy compliance tool, load up the official Government hardening policy in XCCDF for, say, Windows 2003, run a compliance scan, export the data in OVAL, and load the results into a final application that can help your CISO keep track of all the vulnerabilities.  Basically, imagine that you’re DoD and have 1.5 million desktops–how do you manage all of the technical information on those without having tools that can import and export from each other?

And then there was the Federal Desktop Core Configuration (FDCC): OMB and Karen Evans handed SCAP its first trial-by-fire.  FDCC is a configuration standard that is to be rolled out to every Government desktop.  According to responses received by OMB from the departments in the executive branch (see, Karen, I WAS paying attention =)   ), there are roughly 3.5 Million desktops inside the Government.  The only way to manage these desktops is through automation, and SCAP is providing that.

He sings, he dances, that Tony Sager is a great guy: So he’s presented at Black Hat, now SCAP 2008 (.pdf caveat).  Basically, while the NSA has a great red-team (think pen-test) capability, they had a major change of heart and realized, like the rest of the security world (*cough*Ranum*cough*), that while attacking is fun, it isn’t very productive at defending your systems–there is much more work to be done for the defenders, and we need more clueful people doing that.

Vendors are jumping on the bandwagon with both feet: The amount of uptake from the vulnerability and policy compliance vendors is amazing.  I would give numbers of how many are certified, but I literally get a new announcement in my news reader ever week or so.  For vendors, being certified means that you can sell your product to the Government, not being certified means that you get to sit on the bench watching everybody else have all the fun.  The GSA SAIR Smart-Buy Blanket Purchase Agreement sweetens the deal immensely by having your product easily purchasable in massive quantities by the Government.

Where are the rest of the standards: Yes, FDCC is great, but where are the rest of the hardening standards in cute importable XML files, ready to be snarfed into my SCAP-compliant tool?  Truth be told, this is one problem with SCAP right now because everybody has been focusing on FDCC and hasn’t had time yet to look at the other platforms.  Key word is “yet” because it’s happening real soon now, and it’s fairly trivial to convert the already-existing DISA STIGs or CIS Benchmarks into XCCDF.  In fact, Sun was blindsided by somebody who had made some SCAP schemas for their products and they had no idea that anybody was working on it–new content gets added practically daily because of the open-source nature of SCAP.

Changing Government role: This is going to be controversial.  With NVD/CVE, the government became the authoritative source for vulnerabilities.  So far that’s worked pretty well.  With the rest of SCAP, the Government changes roles to be a provider of content and configurations.  If NIST is smart, they’ll stay out of this because they prefer to be in the R&D business and not the operations side of things.  Look for DHS to pick up the role of being a definitions provider.  Government has to be careful here because they could in some instances be competing with companies that sell SCAP-like feed services.  Not a happy spot for either side of the fence.

More information security trickle-down effect: A repeated theme at SCAP 2008 is that the public sector is interested in what Big SCAP can do for them.  The vendors are using SCAP certification as a differentiator for the time being, but expect to see SCAP for security management standards like PCI-DSS, HIPAA, and SOX–to be honest here, though, most of the vendors in this space cut their teeth on these standards, it’s just a matter of legwork to be able to export in SCAP schemas.  Woot, we all win thanks to the magic that is the Government flexing its IT budget dollars!

OS and Applications vendors: these guys are feeling the squeeze of standardization.  On one hand, the smart vendors (Oracle, Microsoft, Sun, Cisco) have people already working with DISA/NSA to help produce the configuration guides, they just have to sit back and let somebody turn the guides into SCAP content.  Some of the applications vendors still haven’t figured out that their software is about to be made obsolete in the Government market because they don’t have the knowledge base to self-certify with FDCC and later OS standards.  With a 3-year lead time required for some of the desktop applications before a feature request (make my junk work with FDCC) makes it into a product release, there had better be some cluebat work going on in the application vendor community.  Adobe, I’m talking to you and Lifecycle ES–if you need help, just call me.

But how about system integrators: Well, for the time being, system integrators have almost a free ride–they just have to deal with FDCC.  There are some of them that have some cool solutions built on the capabilities of SCAP, but for the most part I haven’t seen much movement except for people who do some R&D.  Unfortunately for system integrators, the Federal Acquisition Regulation now requires that anything you sell to the Government be configured IAW the NIST checklists program.  And just how do you think the NIST checklists program will be implemented?  I’ll take SCAP for $5Bazillion, Alex.  Smart sytem integrators will at least keep an eye on SCAP before it blindsides them 6 months from now.

Technical compliance tools are destined to be a commodity: For the longest time, the vulnerability assessment vendors made their reputation by having the best vulnerability signatures.  In order to get true compatibility across products, standardized SCAP feeds means that the pure-play security tools are going to have less things to differentiate themselves from all the other tools and they fall into a commodity market centered on the accuracy of their checks with reduced false positives and negatives.  While it may seem like a joyride for the time being (hey, we just got our ticket to sell to the Gubmint by being SCAP-certified), that will soon turn into frustration as the business model changes and the margins get smaller.  Smart vendors will figure out ways to differentiate themselves and will survive, the others will not.

Which leads me to this: Why is it that SCAP only applies to security tools?  I mean, seriously, guys like BigFix and NetIQ have crossover from technical policy compliance to network management systems–CPE in particular.  What we need is a similar effort applied to network and data center tools.  And don’t point me at SNMP, I’m talking rich data.  =)  On a positive note, expect some of the security pure-play tools to be bought up and incorporated into enterprise suites if they aren’t already.

Side notes:

I love how the many deer (well over 9000 deer on the NIST campus) all have ear tags.  It brings up all sorts of scientific studies ideas.  But apparently the deer are on birth control shots or something….

Former Potomac Forum students:  Whattayaknow, I met some of our former students who are probably reading this right now because I pimped out my blog probably too aggressively.  =)  Hi Shawn, Marc, and Bob!

Old friends:  Wow, I found some of them, too.  Hi Jess, Walid, Chris, and a cast of thousands.

Deer on NIST Gaithersburg Campus photo by Chucka_NC.

Bookmark to:

Hide Sites

Just a quick slideshow I threw together for a class on making a Backtrack 3 bootable USB drive.  I sanitized it for public use, but I figure there’s some reusable content that somebody will thank me for later.

Bookmark to:

Hide Sites