Towards Actionable Metrics

Posted March 4th, 2008 by

Ah yes, our favorite part of FISMA:  the ongoing reporting of metrics to OMB.  Last year’s guidance on what to report is in OMB Memo 07-19.  It’s worth the time to read, and you probably won’t follow with the rest of this blog post if you don’t at least skim it to find out what kind of items get reported.

 Still haven’t read it?  Fer chrissakes, just look at pages 24-28, it’s a fast read.

If you look through the data that OMB wants, there are 2 recurring themes:  What is the scope/extent/size of your IT systems, and how well are you doing what we told you to do to protect them?  In other words, how effectively are you, Mr CISO, executing at the operational level?

We’re missing one crucial bit of process here–what are we actually going to do with scoping metrics and operational performance metrics at the national, strategic level?  What we are collecting and reporting are primarily operational-level metrics that any good CISO should at least know or be able to guess at to do their job, but it’s not really the type of metrics that we need to be collecting at levels above the CISO unless our sole purpose is to watch over their shoulder.

As our metrics gurus will point out, the following are characteristics of good metrics:

  • Easy to collect:  I think the metrics that OMB is asking for are fairly easy to collect now that people know what to expect.  Originally, they were not.
  • Objective:  Um, I’ll intentionally side-step this one.  Suffice it to say that I’ve heard from several people a story where the punch line goes something like “Your security can’t be this good, we’ve already decided that you’re getting a “D”.
  • Consistent:  Our consistency is inconsistent.  Look at how many times the FISMA grading scale has changed, and we still wonder why people think it’s not rooted in any kind of reality.  And yes, I’m advocating yet another change, so I’m probably more an accomplice than not.
  • Relevant:  We do a fairly good job at this.  Scoping and performance metrics are fairly relevant.  I have some questions about if our metrics are relevant at the appropriate level, but I’ve already mentioned that.
  • Actionable:  This is where I think we fall apart because we’re collecting metrics that we’re not really using for anything.  More on this later….

Now, as Dan Geer says in his outstanding metrics tutorial, the key to metrics is to start measuring anything you can (caveat, 6-MB PDF).  The line of though goes that if you can collect a preliminary set of data and do some analysis on it, it will tell you where you really need to be collecting metrics.

The techie version of this is that the first server install you do, you will blow it away in 6 months because you now know better how you operate and what you need the configuration really to be.

Now ain’t that special?  =)

So the question I pose is this:  after 6 years, have we reached the watershed point where we’ve outgrown our initial set of metrics and are ready to tailor our metrics based on what we now know?

I think the answer is yes, and applying our criteria for good metrics, what we need to answer is a good set of questions:

  • What national-level programs can reduce the aggregate risk to the government?
  • What additional support do the agencies need and how do we translate that into policy?
  • As an executive branch, are we spending too much or too little on security?  Yes, I know what the analysts say, but their model is for companies, not the Government.
  • What additional threats are there to government information and missions?  Yes, I’m talking about state-sponsored hacking and some of the other things specific to the government.  Is it cost-effective to blackhole IP ranges for some countries for some services?
  • Is it more cost-effective to convert all the agencies to one single NSM/SIEM/$foo ala Einstein or is it better to do it on a per-agency basis?
  • What is the cost of implementing FDCC, and is it more cost-effective and risk-effective to do it immediately or to wait until the next tech refresh on desktops as we migrate to Vista or upgrade Vista to the next major service pack?
  • What is the cost-benefit-risk comparison for the Trusted Internet Connections initiative, and why did we come up with 50 as a number v/s 10 or 100?
  • Is there a common theme in unmitigated vulnerabilities (long-term, recurring POA&Ms) across all the agencies that can be “fixed” with policy and funding at the national level?  Say, for example, the fact that many systems don’t have a decent backup site, so why not a federal-level DR “Hotel”?
  • Many more that are above and beyond my ability to generate today…

In other words, I want to see metrics that produce action or at least steer us to where we need to be.  I’ve said it before, I’ll say it again:  metrics without actionability means that what we’ve ended up doing is performing information security management through public shame.  Yes, some of that is necessary to serve as a catalyst to generate public support which generates Congressional support which gets the laws on the books to initiate action, but do we still need it now that we have those pieces in place?

If I had my druthers, this is what I would like to see happen, maybe one day I’ll get somebody’s attention:

  • OMB and GAO directly engage Mr Jacquith to help them build a national-level metrics program.
  • We produce metrics that are actionable.
  • We find a way to say what our problems are without overreacting.  I don’t know if this can happen because of cultural issues.
  • We share the metrics and the corresponding results with the information security management world because we’ve just generated the largest-scale metrics program ever. 

And oh yeah, while I’m making wishes, I want a friggin’ pony for Christmas! =)

Similar Posts:

Posted in FISMA, What Doesn't Work, What Works | 3 Comments »

3 Responses

  1.  Halon73 Says:

    As always; Brilliant and yet a continuing source of amusement. We can have all the metrics in the world but as you mentioned we have to have a culture shift; but this must happen first. Unless we have support from the top down and back up to the top for change (Obama) then I think nothing will actually happen.

    I have taken your post to heart and in my own reporting demand that when we report to management that we only report actionable items that our GS manager can do something about. Otherwise charts and graphs are just meaningless colors. Yes tracking trends is important but what is a trend but a result of actions that came from reports that supported managers being managed effectively.

    I’ve come to understand that customers, managers, and anyone that is not in Information Security needs to be managed covertly and tasking folks by controlling them with covert mind control is the solution.

    It’s not enough to answer the mail but to do so in a way produces a ripple effect result.

    Metrics must first task our managers and customers to think and do something with lasting ripples and yes waves of change.

    A report is dead on arrival if it just gets filed away or is a door stop in some GS’ers office for the day when the IG comes. We’ve seen millions of pages generated because of FISMA that is useless but for only to make the Dept IG “go away” until next year.

    So rather than labor over the details of metrics (while important) lets start the culture shift NOW by changing the foundation of reporting TODAY. I call on all of my brothers and sisters in INFOSEC to stop delivering “pretty pictures” and raw data dumps. The folks who you are sending these things to don’t know what they are looking at and won’t use them to do anything by tack them on a wall someplace to make it look like they are actually working.

    Instead frame your reporting in such a way that they have to “do” something to make the report complete. No, don’t make it an approve or disapprove check box but an interactive process that says “HELLO! I need you to do X on this action that WILL be tracked and continuously monitored in this report until information security is satisfied that X has been done and we can move on.”

    A year ago I thought I was the servant to a master but I realize that I am the master of the master and that I need to manage my manager by tasking them to complete X by saying that if they don’t participate then X will never go away and customer “Z” will never improve the security of the system.

    The indifference and laziness of the American society infects all dimensions of culture with the impact that all but a small minority of workers goal is to only make it to the COB with as little effort expended as possible. We must keep this in mind and hold these people accountable because the expectation on information security is “thats your problem” which is risk adverse and means that customer “Z” doesn’t have to do anything but be disconnected.

    In order for us to achieve real and tangible security we must not only connect the systems and close the holes but connect people on every level possible to engage them in maintaining the security of the data they process!

    Cheers – PZ

  2.  The Guerilla CISO » Blog Archive » Ack! With the Mandates! Says:

    […] effective security.  Of course, I think it’s well-written because it says some of the same ideas that I’ve been saying for awhile now.   […]

  3.  Russell Thomas Says:

    Wonderful post! Clear and spot-on!

    The best way to evaluate *any* metrics program is to ask: “What decisions and actions do these metrics influence?” and “Do the metrics provide sufficient information to motivate action?”

    As you say, the “action” or “decision” may be at any of several levels — from operational to strategic. The designers of metrics systems *must* think through how their metrics will be used.

    (BTW, could you add to your Blogroll?)


Leave a Comment

Please note: Comment moderation is enabled and may delay your comment. There is no need to resubmit your comment.

Visitor Geolocationing Widget: