IT observability tool proliferation fuels AIOps deployments

Observability is the management strategy that prioritizes the issues most critical to the flow of operations. It's a necessary practice for businesses in the throes of digital transformation, migrating to the cloud, securing their processes and speeding their IT operations and applications development environments. Businesses practicing observability are increasingly integrating AIOps to improve automation processes, increase end-to-end visibility, provide real-time data analysis and yield trustworthy outcomes.

To gauge the growing importance of observability and increasing deployment of AIOps tools, Jon Brown, senior analyst for IT operations and sustainability at TechTarget's Enterprise Strategy Group, conducted an in-depth survey of 374 IT, applications development, software engineering and DevOps professionals at midmarket and enterprise organizations in the U.S. and Canada. For this report, "Distributed Cloud Series: Observability and Demystifying AIOps," survey respondents were asked about operational management observability, their issues and concerns, and the adoption of observability and AIOps tools.

In this interview, Brown discusses the report's key findings, unexpected surprises, the importance of AIOps and advice for businesses deploying and planning to deploy AIOps tools as part of their observability initiatives.

What does the report tell us about the role of observability and AIOps in business?

Jon Brown

Jon Brown: Observability is the way that we're going to be managing our IT infrastructures. And we're going to increasingly do it with the aid of AIOps in order to scale. The observability market is hot, and it is evolving quickly. We're still seeing new vendors coming in, and we see very strong purchase intent from our audience.

Midmarket companies are typically the last to embrace technology, and we have a bunch of reasons why we think that. Mostly it's because they have weak CIOs -- if they have CIOs at all. They typically are not in charge of their budgets. CEOs are still involved in budgeting in a midsize company in a way that they're not in a large company.

Observability is getting to the point that it makes sense in the midmarket. That is an indicator of maturity of the tools as well as the maturity of cloud and cloud-native technology.

But there are still a lot of challenges. This is very much a work in process. They are still tooling their observability infrastructures, and it's collinear with both their digital transformations and their evolution to cloud-native. And once you go cloud-native, you need observability because what's happening with your apps is so ephemeral. They're not running all the time. You don't know how many instances are running.

Observability helps you scale your ability to manage more complex and cloud-native infrastructures. Organizations that have advanced observability practices spend more time doing value-added type of work -- innovating as opposed to maintaining. That is a positive outcome.

How does AIOps fit into observability?

Brown: AI is automation with a brain. AIOps is leveraging the power of artificial intelligence to correlate the phenomena that you see in your observability or operational data. It's using an accumulated knowledge base to determine the root cause of a problem and to either suggest or actually implement and make a change in a system.

Many observability vendors are adding AIOps capabilities to their observability platforms or tools. And the reason is that observability throws off a ridiculous amount of data. And as you get more and more cloud-native, you actually get more and more data. It's not possible for a human being to interpret all of that data. And it gets worse because you may have monitoring and observability at different levels of the stack.

We have all this telemetry data, 11 different points of data for every moment of every application. What's important to save and what isn't? And that's another area where you know AI can help you out -- machine learning, especially. It's a machine learning function, basically, and looking at analyzing, predicting outcomes or giving whatever answers they're looking for. Can we filter through all this stuff? Can you tell me what's real, what is important and what I can ignore?

What's driving AIOps tool deployments?

Brown: The number one thing that's happening in IT is acceleration of operations -- digitizing business processes. Once we digitize them, the idea is, 'OK, crank up the speed,' right? We're automating this stuff. We should be able to have greater throughput. So that's the need for speed, and the increasing pace of IT operations is what is driving this investment in observability and AI.

Second is the desire again to relate spend and IT effort back to business outcomes. And for the first time ever, we can do that because if you have a wholly digitized process, you actually can observe every single step along the way from every perspective. You can see how much it cost, how much resource it used and how much revenue it generated.

And this is new and different, and it's the Holy Grail. You know what we're really doing here is we're creating a new business application called financial operations management. It's exactly like ERP for your factory, except it's for your compute infrastructure. We are effectively reinventing ERP when it comes to IT operations management because our factory now is our IT infrastructure.

What are IT and developer teams saying about AIOps?

Brown: The demand is strong. Only 8% say we have no plans to use AIOps, so you know from a maturity standpoint that tells us everybody knows it is happening. Just under half say that AIOps is a positive thing and that it helps us scale our observability practice. And that's the point here, right? We're getting more and more data. No humans can read it. We need AIOps to interpret the stuff and do stuff for us.

But another 28% are saying it's had absolutely no impact whatsoever, and then another 27% say the impact has been negative. This is a sign of an early market. We don't really know what to do with this yet. That tells the story of the state of AIOps, which is adolescence.

AI adoption is happening, and it's on the horizon for everybody. It's going to become table stakes five years from now. But we are in the part of the adoption curve where the expectations for the technology versus the reality of the technology is not there yet. It's likely that we need a couple of turns of the maturity wheel before this becomes consumed -- a packaged product that you can use out of the box that adds value immediately.

Which findings in the report surprised you the most?

Brown: We definitely noticed some divisions between IT operations and developers in how they're interpreting the use cases and benefits of these new tools based on their roles. That split was an eye-opener. They're not a little bit apart; they're on separate planets when it comes to AIOps, and AIOps is at the bleeding edge of what's happening in observability. So, it is super interesting to see that split. The developers are not the primary target of observability, but they're actually getting the most benefit.

Developers tend to find IT operations kind of a hassle. They're much more bullish on AIOps and what it can do for them. Developers are nine times more likely to say this is a very, very, very important function of observability and of our practice. They need as much automation as possible. They're not in the IT ops business. AIOps can build my infrastructure for me and fix my infrastructure problems without having to put a service desk ticket in. That's a good thing. And so, from a developer standpoint, AIOps solves a big problem for them, which is dealing with operations.

IT ops' job is ops. They're a little less enthusiastic and a little more conservative about AI because they know that the root of cascading failures is to make one mistake. They're very nervous about allowing AI to actually do their job. Every IT ops person I've ever spoken to says we love AIOps. We don't trust it, but we love it because it tells us what to do. It's a great recommendation engine. From a maturity standpoint and a product readiness standpoint, IT doesn't think it's there yet.

How would you advise organizations planning to add AIOps to their observability initiatives?

Brown: Look at the AI capabilities and pick a few. Narrow your focus. Treat this like a science project. Pick two or three use cases that make sense. And do those.

Don't think of this as all-encompassing AI. What you have are tools that can help you do specific things better. Given the current technology's maturity level, start developing plans on how you're going to automate and have AI do more and more of the especially repetitive business processes or services that you deliver.

Also, make sure you involve your development team and development leadership in your tool selection. Make sure it's going to work for them and fit into their workflow. They are going to be more intimately involved in this than probably any other team.

So, my recommendation would be to make sure you're collecting and understanding the issues that the developers are having so that their concerns can be reflected in your purchase. If you don't, you're not going to get the full benefits of what you're trying to do. If you're not involving DevOps in your IT ops decisions, you might be missing some valuable input that is going to cost you money.

From the vendor standpoint, keep at it. Continue to improve and invest in AI because this is going to be the secret to your success going forward. The second thing is to make it easier for your buyers to extract value from AI. Really focus on some specific high-value use cases, like helping users with their data, their observability, their data sprawl. You're thinking that you're only solving IT operations problems. But, in reality, you're also solving DevOps problems. So, make sure that DevOps is one of your key customers.

Editor's note: This interview has been edited for clarity and length.