I have been giving the idea of metrics considerable thought lately.  I’ve had a few people tell me that they have been asked to “justify what they do” by their management.  In a way that is understandable.  In tough economic times a business should examine everything they do and determine what the value added tasks are and what they aren’t.  As long as this is a question being asked of every department then we shouldn’t be surprised.  Unfortunately it seems that those involved in information risk management get asked this question way too often. 

It is very hard to quantify information risk management it is important to find some way to determine if you are getting ahead or if you are falling behind.  So rather than give the classic “it depends” response I will instead offer some considerations that you should take when considering the appropriate metrics to generate and report. 

Metrics are a means to measure the performance of a particular area.  They are intended to provide the feedback necessary to support continuous improvement.  Information Security suffers from the fact that metric development can often neglect the larger picture therefore lessening their ability to influence key decision makers within the organization outside of the technical environment. 

 This issue stems from the ability to measure so many different aspects of our environments.  While this ability is beneficial, it can also be a liability if the information generated cannot be translated beyond the information technology department.  Hopefully this post will help with addressing these concerns and provide issues to consider when information security professionals develop metrics for senior management.  

Answer the “Why” before the “What” and “How”

Information Security (Infosec) metrics must be based not only on the goals and objectives of the Infosec department but must also be directly tied to an organizations overall goals and objectives.  The purpose of Infosec is to support and facilitate business through the control of information.  This supporting role requires that all Infosec initiatives relate to the overall mission of the organization. 

The business case for all initiatives must directly relate to and support the overall mission of a company.  This linkage should also form the basis of all metrics developed within an organization.  These linkages dictate the desired result and identify critical areas to measure.  This measurement must yield quantifiable information that can be used to compare current performance with past performance for analysis.  This information must also be easily obtainable as the burden of collecting the information should not exceed the benefit gained from its collection. 

The first question that must be asked is “Why do we need to develop a metric?”  Metrics must serve a purpose beyond themselves and be relevant to their environment.  Metrics can assist decision makes with the isolation of problems, the generation of data to justify investments requests and the need to ensure that funded investments are returning value from the resources committed.  Metrics can assist companies in their ability to demonstrate compliance with the applicable laws, rules, and regulations of a particular industry.  Care must be taken though to keep the number of metrics manageable as too many metrics can lead to the dilution of critical information. 

“What” needs to be measured?

Once we have a handle on why we need to measure, we can then focus on what we need to measure.  This requires prioritization on the part of the Infosec professional.  So many different elements of our environments can be measured that we need to take care that information overload does not occur. 

Let us examine some areas that Infosec typically addresses:

• Identity Management,
• Change Control and Configuration Management, and
• Contingency Planning and Incident Response.

These areas give us a basis to examine the considerations that we need to take into account when developing metrics for these areas.

 Identity Management (IM)

Identity Management includes those areas that are typically involved in the identification of users and devices within an environment as well as the ability control access to that environment and the information therein.  While IM lends itself to many different metrics that are useful from an operational standpoint, senior management often does not need this level of detail.  The idea is to provide information that is relevant to senior management and this information should be related to the benefit that the organization is receiving through the use of the technology.  Metrics that may be useful could include:

•  The Number of Unauthorized Access Attempts vs. the Total Number of Users/Devices Accessing the Network, and
• The Number of Unauthorized Access Attempts to Critical Data Storage Areas.

This information provides senior management with a measure of success of not only security controls but with user training and help desk support with regard to new identity Management projects.  These metric measures put IM projects into perspective and highlight their effectiveness. 

 Change Control and Configuration Management

Change Control and Configuration Management involves all aspects of change within the environment and the impact these changes have on Infosec.  These aspects involve the systematic proposal, justification, implementation, test/evaluation, review, and disposition of changes to the information system(s), including upgrades and modifications.  

The metrics involved in this aspect of Infosec can be overwhelming.  This is an area where the “big picture” view that senior management needs can also be very useful at the operational level as well.  Some metrics to consider:

• Number of Required Functional Change Requests
• Number of Required Security Change Requests
• Number of Feature (Non-required) Requests
• Average LOE (Man-hours and funds) for each of the above
• Average projected loss of user productivity for each of the above

These metrics can be used to place the Change Control and Configuration Management initiatives into perspective.  Care must be taken though to relate these metrics directly to the company’s overall mission.  This linkage will vary depending upon the organization and how each department interacts. 

Contingency Planning and Incident Response (CPIR)

Contingency Planning and Incident Response addresses those areas critical to maintaining the operational ability of an organization during an adverse event.  These events need to be related to the loss the organization anticipates experiencing (or is experiencing in the case of Incident Response).  CPIR is traditionally one of those areas seen as a “money pit” within an organization.  This perspective gains strength the longer a company continues without a need to take CPIR actions.  Proper measurement and alignment of CPIR initiatives to an organization’s mission are critical.  While many metrics can be developed around this subject area, the key is to justify the need for resources.  To this end, it is suggested that a breakdown of activities include the cost of CPIR activities per hour vs. the projected expected loss per hour for that activity. 

Conclusion

The aim of metrics is to measure a particular activity in order to generate feedback that is needed in order to effect change that results in an improvement of the activity.  To that end there are many different metrics that can be useful to various levels within an organization.  This makes it important to aim the metric at the appropriate audience in order to present that information that is most relevant.  As with any presentation or proposal, understanding the intended audience is key to achieving success.  This is no less important with the development of metrics.

Over the years I have typically seen metrics that mean something to the network/IA staff but not to senior management.

When these metrics are given to C-Level executives they glass over and don’t really understand what they are being told. They rapidly try to figure out if a larger or smaller number is bad so they can attempt to put what they are given in some sort of context. The problem is that they become use to this and begin to rely on these numbers without really understanding what the numbers are really telling them. They think they understand but they don’t and as a result don’t give the priority to IA in the way they should if they really understood what they were being told.

We, as IA professionals, become complacent with this arrangement because that is what we are use to and our executives appear to understand. We confuse acceptance for understanding.

As IA professionals, we often only answer half of the question and totally forget about the business side of the equation. We need to understand our audience and frame our metrics in terms that they understand rather in terms that we understand. We need to find a way to both qualitatively and quantatively measure our success and translate that into business speak each and every time.

Think about it this way – we can create the business case for what we do each and every time we generate metrics without having to continuously try to convince others.

I believe it is important to measure what our management wants to see, it is equally important that we tell them what they need to measure. Now this may seem straightforward but I don’t just mean giving them answers to questions they haven’t asked and then trying to convince them that this information is important. What I mean is that we aim to provide the appropriate metrics with every project proposal we submit to management for funding. 

That would mean not just telling management that we want to be able to correlate and filter x number of log entries from y number of devices but that we want to be able to say reduce the number of false positive incidents by 25% resulting in a savings of z dollars per year by implementing a SEIM project.

As with most of our IA duties, the qualitative nature of our job makes the development of useful metrics often difficult. Yes we can calculate the number of viruses our desktop AV catches a week but is that really meaningful in the long run? It may be if we can tie that number into a formula that calculates response, recovery, and lost productivity numbers per virus outbreak.

I guess what I’m saying is that the actual measurement isn’t as important as constantly linking that information with figures that mean something to management. This is probably in line with Deming’s emphasis on controlling the process as it ties the process into terms that resonate in business terms.

Thoughts?