Modern IT KPIs emphasize cloud, DevOps and user experience

Technical drivers

The rise of cloud and DevOps has transformed IT in many ways -- including KPIs.

"DevOps has a unique set of KPIs in terms of how applications are developed, how they are provisioned, how they are maintained, how they operate, how frequently they may fail or need to change, and how quickly they can be fixed," said Carl Lehmann, principal analyst at 451 Research. These unique DevOps KPIs include metrics such as time to provision, time to upgrade and time to value.

Additional DevOps metrics include deployment frequency, which tracks code deployment to relay how quickly new features roll out and to alert DevOps teams to possible bottlenecks. Feature prioritization, another example metric, measures how often end users consume an offered service, which enables DevOps teams to prioritize the capabilities users value the most.

Traditionally, many KPIs for IT operations -- such as mean time to repair -- have been rooted in IT service management and, more specifically, the ITIL framework. And while these KPIs are still relevant, the IT industry's shift from a data center to a cloud-centric model has also changed the way teams manage and assess performance.

Carl Lehmann

"Cloud-native architecture is the way companies are pursuing their IT strategy," Lehmann said. "They're measuring performance based on how well their cloud service provider operates and how well they're using a cloud service operator."

Cloud computing services follow a shared responsibility model between providers and their customers. With infrastructure as a service, for example, cloud providers are responsible for the underlying hardware, and enterprises are responsible for the applications that run on top of that hardware. For IT teams, this shift emphasizes metrics related to cloud service availability, which ensure applications meet user expectations.

Organizations should also track metrics such as return on investment to ensure they get value from cloud-based IT assets, Lehman said.

A business -- and user -- focus

In addition to cloud and DevOps adoption, IT's role as a service provider to both internal and external users also prompts changes to KPIs. More specifically, IT metrics need to account for business goals.

"You have to accept this notion that what IT does has to be aligned with the outcomes of the business, not just with the weather inside the IT organization," said Joseph Pucciarelli, group vice president and IT executive advisor at IDC.

As such, results-oriented metrics, such as those related to end-user and customer satisfaction, should replace, or at least supplement, some of the more traditional, process-focused IT metrics tied to time and cost.

A process-centric KPI for IT, for example, would be the rate at which help desk tickets are closed. This metric inspires speed in help desk technicians, but doesn't always satisfy users, especially if the issue isn't resolved correctly and they have to reopen a ticket. A more results-oriented KPI, in this example, would measure users' experience with the help desk ticket process.

Joseph Pucciarelli

"It's [about] the satisfaction with the service the organization provided," Pucciarelli said. "That's an example of aligning with outcomes."

IT admins should emphasize reliability and performance as it relates to user and customer satisfaction, agreed Brian Kirsch, IT architect and TechTarget contributor. An application stack -- especially one that spans the cloud and on premises -- has a lot of moving pieces. Instead of diving right into the details of each component, start with the overall picture of the user experience and work backward to identify technical ways to improve it. In other words, begin with the forest, then look at the trees.

To measure application downtime in the context of the user experience, IT teams should remember that not all downtime is created equal, Pucciarelli said. For example, application downtime for an app that has 100 users will have steeper consequences than one that has 10, even if the outage length is the same for both. To more accurately measure customer satisfaction, look at metrics such as user outage minutes, which reflects when an organization fails to meet its users' expectations.

A change in KPIs requires a change in culture

The evolution of KPIs for IT operations brings with it a challenging culture change.

"When we talk with most executives, most CIOs will say, 'Okay, this makes sense,'" said Pucciarelli, about adopting results-oriented KPIs. "But [for] the folks who report to the CIO, this is a heretical approach because they grew up measuring the efficiency of their processes. So how many help desk tickets did I close? What's the percentage of downtime? You know, process-centric KPIs to the IT organization."

Additionally, KPIs related to user satisfaction can be more difficult to measure than traditional metrics related to time or cost. Communication between IT teams and end users needs to increase if enterprises want a reliable picture of the user experience.

Brian Kirsch

"People are only vocal when there's an issue. You can have 100 people and 99 are doing well -- you're only going to hear from the one that's complaining up a storm," Kirsch said. Without a mix of negative and positive feedback, IT teams only see a limited picture of what is happening. Try to collect thorough feedback through user surveys and similar tools.

"KPIs are useless if you don't have any closed loop on what's going on," Kirsch said.

For some organizations, a shift from process-centric to results-centric IT KPIs will only come at the heels of a management change.

"Some of the organizations that have older IT management, they're stuck in that realm -- the nine to five [o'clock], everyone wears a tie," said Kirsch. "And as that management changes, you're starting to see more flexibility [and more] looking at the soft numbers, understanding that they have an impact."

Small and mid-sized companies are typically more flexible and able to adjust faster than larger enterprises, he added.