There are a lot of specific, technical KPIs for IT admins to track, but what are the nontechnical signs that your organization's DevOps transformation is going well?
If you ask a C-level executive the value of DevOps, the most common answer would be the promotion of business agility. Software projects must be responsive because the business processes they support need to respond to problems and opportunities. The best metrics to judge DevOps success assess its impact on business agility. This means DevOps teams must turn their focus from inside the projects to inside the project's drivers.
Before IT organizations assess nontechnical measures for DevOps success, they must establish what DevOps will contribute to the business. When an IT organization, the CIO and the CFO agree on DevOps adoption, the agreement should be set to improve software developments and operations.
Those improvements must be identified -- and described as precisely as possible -- to create a standard reference IT teams can apply to the possible nontechnical measures of DevOps success.
Metric #1: Mean time to response
The best starting point is mean time to response. Forget how long it takes to do a commit or a pull and ask how long it took the DevOps project-reliant business processes to respond to business problems and opportunities.
Time to response should measure against both IT organization estimates and line department goals, with emphasis on the latter. As project requests and project completion are easily assigned specific dates, mean time to response is an objective but nontechnical metric.
The biggest complaint line organizations voice on their relationship with IT is the amount of time it takes to address and deliver project requests. In almost every enterprise, IT is essentially a staff function, an expense justified by its business operations support. Delays in line projects that seek to respond to issues and opportunities could result in business losses.
If DevOps can improve time-to-response IT performance, that goes a long way to prove it's successful. Tracking projects from request to realization can provide quantitative and qualitative data to validate that claim.
Metric #2: Business case progress
A related DevOps success measurement is business case progress. Nearly all IT projects, including those that involve DevOps, require a definable business case, where ROI compares to company standards for project returns. The percentage of projects that make the business case is a valuable metric to gauge DevOps effectiveness.
The majority of enterprises require a specific business case to validate IT spending. Every significant IT project goes through this review, and most companies require a post-project review of the business case as well, to validate early projections against reality.
If DevOps is being used successfully, the number of cases where an IT project failed to make the business case -- for reasons such as delays in execution, problems with software quality or other development-related reasons -- will decline. Normally the CFO organization has this information available and uses it to assess the value of DevOps.
The true measure of success for an IT project with a goal to improve business operations is whether it does, in fact, result in improvement.
Metric #3: Postmortem assessments
Line feedback is important, too, and the best way to obtain it is via postmortem satisfaction assessments, where compliments and complaints of line personnel involved with the project outcomes are solicited. It's important to structure these assessments, rather than collect comments extemporaneously, or organizations could miss important information.
Metric #4: Business case protection
A surprisingly large number of IT projects fail to gather this information. The true measure of success for an IT project with a goal to improve business operations is whether it does, in fact, result in improvement.
Part of the reason is that most IT projects fail to address a critical requirement, i.e., the need to protect the business case. At the planning level, IT projects should identify what specific features or behaviors are required of the projects' applications to meet the business case through a cooperative discussion between line organizations and IT. That planning step is essential to the satisfaction assessment.
The line personnel involved in the planning should, on completion of the project, solicit comments from their own organizations and feed those comments back to IT. Project success is indicated by two things: First, the net of the positive and negative comments received, and second, the number of comments that reflect threats to the application's success score in meeting business requirements. Together, these comments reflect how DevOps' role in the project is then a measure of DevOps success.
Metric #5: Post-production state evaluation
The final success measurement strategy is to evaluate post-production-phase do-overs. Users and IT staff alike are sometimes reluctant to accept comments and complaints as a measure of success because they are necessarily subjective. If IT teams can't address biases, then look at how many projects had post-production revisions or complete reworks, for reasons DevOps should have resolved.
Both line and IT organizations agree that a serious project problem will require a second project to fix it, and that's a decision that is rarely subjective. If DevOps can reduce the number of project revisions and remakes, then the cost savings attributable to that reduction is a mark in the gains column of DevOps effectiveness.
Nontechnical measures of DevOps success aren't just an exercise in organizational political correctness. IT organizations often focus on their own productivity and not on the impact of their projects on business productivity. The techniques described above will get DevOps activity focused where it should be -- on the business.
