How to perform a cost/benefit analysis: Build a model, not a number

I have something that needs to stay just between you and me.

As IT professionals, you and I need to admit to ourselves and to each other that our current methods for evaluating projects are not only pretty much a waste of time, but they also sometimes lead us to bad decisions.

The Real Niel
Niel Nickolaisen

The methods I am talking about are traditional cost/benefit analysis. Most of us use some type of cost/benefit analysis to decide which projects to fund and then how to rank the projects. These analyses include internal rate of return (IRR), net present value (NPV) and payback period. The dirty little secret of these methods is that while we might be able to reasonably estimate project costs, we usually have no real clue about the benefits and so have no real hope of deriving a precise number to use in our IRR or NPV calculation. This is especially true of projects intended to generate revenue.

So the uninitiated take a guess as to the costs and benefits and see whether their projects get approved. Those of us who are more savvy and cynical game the system and estimate the benefits required to compensate for enough of the costs to meet the IRR or NPV threshold.

My frustration with this process is twofold: First, costs and benefits often do not capture all of the things we should consider in making a decision. Second, this approach forces us to try to make decisions before we have learned enough to do so successfully.

For the past couple of years, I have been experimenting with some changes to the traditional cost/benefit analysis that seek to overcome these two weaknesses. The result is a decision model in place of cost/benefit. I build a distinct decision model for each major project, but my model usually includes three inputs:

• Costs
• Benefits
• Considerations

We are familiar with how to capture costs, and so my model doesn't change this input unless we don't yet know enough to capture costs (more on that in a minute). In my models, considerations such as time, risks and flexibility carry a significant weight. For example, if I miss my target window, how do the benefits deteriorate? If I don't do this project, am I increasing risks? Is this project an enabler of long-term plans? If so, these considerations might be my most important decision input.

On the benefits side of life, rather than trying to predict a number, we need to recognize that estimated benefits are highly uncertain. Even if we are good at guessing, the benefits are often beyond our control. When we implement a new warehouse management system, can we force the manufacturing/distribution function to change its processes enough to realize the benefits? If so, tell me how, because I have never been able to do so. My decision models recognize that we should learn about benefits as we go along. So I break projects into pieces that do two things: first, deliver some intermediate, yet measurable value; second, allow me to better quantify the costs, considerations and benefits.

For example, the chief marketing officer came to me with an interesting idea. If his idea worked, we would be able to dramatically improve customer retention. Together, we started work on the traditional cost/benefit analysis. I felt we could get to about the 80% confidence level on the costs, but the benefits were as nebulous as they could be. In order to get to a benefit number, we would need to know the lifetime value of a retained customer. We would also have to know by what amount this new program would increase the number of retained customers and the increased retention length. The best we could do was a wild guess.

So, rather than build a number, we built a model. With the end goal of customer retention, we asked ourselves what we could learn, in stages, to discover whether or not we were achieving the desired benefits. Taking this approach, we established intermediate deliverables and benefits. The first phase was to segment our customers. Once segmented, we could measure the segments for baseline retention information. In this first phase, we achieved benefits of improved direct marketing. Rather than using a scattershot approach to our catalog, television, radio, print and Internet advertising, we aligned our advertising with each segment. We reduced costs, increased sales and also gathered the information we needed to better define the costs, benefits and considerations of the next phase.

Our IT steering committee loved the idea of learning as we went. Not only did the piecemeal approach reduce risk, but it also gave the group the opportunity to kill the project if our decision model about the next phase did not prove accurate.

Traditional cost/benefit works well for straightforward projects, but as costs or benefits become less precise and considerations become more important, I have found that building a model leads to better decisions.

Niel Nickolaisen is CIO at Western Governors University in Salt Lake City. He is a frequent speaker, presenter and writer on IT's dual role enabling strategy and delivering operational excellence. Write to him at [email protected].