Continuous intelligence a trend on the rise

Continuous intelligence is becoming increasingly important for enterprises as they seek a competitive edge.

Often incorporating data streaming, it's the concept of analyzing and acting on data at a steady, high-speed rate. Continuous intelligence is a popular way to apply real-time analytics, and enables organizations to get a clear picture of any situation based on data in near real time rather than days or even weeks after the fact.  

As organizations see things as they happen, they're subsequently able to quickly make data-driven decisions.

With many organizations struggling to survive as economies around the world suffer because of the COVID-19 pandemic, continuous intelligence is more critical now than ever before. Analytics has helped some businesses recognize opportunities to pivot in order to stay viable during the pandemic, and others learn to be more efficient. Using data in near real time has enabled enterprises to react quickly to fast-changing circumstances.

Kevin Petrie, vice president of research at Eckerson Group, has written extensively on continuous intelligence, including a blog post entitled "Five Steps to Succeed with Your Continuous Intelligence Strategy" published on Nov. 11.

Petrie recently took time to talk about the topic, including when enterprises began incorporating data streaming into their analytics, its rise in popularity since then and how it's helped businesses cope during the pandemic.

Just to get started, how do you define continuous intelligence?

Kevin Petrie

Kevin Petrie: Continuous intelligence can take several different forms, but broadly speaking it's a set of functionalities that applies both real-time and historical analytics to core system data and external data and then applies the findings to either recommendations or specific operational actions.

What does continuous intelligence enable an organization to do that it can't when it's not capturing data in real time?

Petrie: Many, many workflows today have very low latency, and the value of data and events is very perishable, so there are short windows of time in which you can react to business opportunities. Continuous intelligence can enable you to very quickly identify a cross-selling opportunity, or an e-commerce site can identify the need for preventative maintenance before a factory part breaks down or a delivery truck breaks down. There are many different cases both in the physical and virtual world in which you need quick action and quick thinking. Continuous intelligence can help you address that.

There are many different cases both in the physical and virtual world in which you need quick action and quick thinking. Continuous intelligence can help you address that.

You mentioned a couple of examples -- can you take me through a use case for continuous intelligence?

Petrie: Broadly speaking, continuous intelligence is often used to improve existing processes for IT operations, for security and for DevOps, but I think the most instructive examples would relate to customer engagement and the cross-selling example. There could be a case in which you have an individual who comes to a website, and based on their browsing history you know they have certain preferences or interests in a certain kind of apparel, and you also know for their purchase history that they periodically can make very large purchases. You can, based on social media trends and based on local weather patterns, start to triangulate exactly what type of cross-selling or real-time offer might be most appealing to that online shopper and generate that before they leave the site before they go to check their online news feed.

That's one of many examples in which you can bring together core transactional information that's traditional structured information in your own purchase database, correlate that to broader trends divined from social media -- unstructured data pieces -- and make a real-time offer to that customer.

Are the decisions based on continuous intelligence generally automated, as in the example you just gave, or does data streaming inform human decision-making as well?

Petrie: It doesn't have to be automated. The output of continuous intelligence can either be an automated recommendation to a human who makes a decision -- it might be someone managing a fleet of delivery trucks who receives an automated recommendation to reroute traffic around a storm coming in, or it might be a specific automated action such as what I talked about before in the example of an automatic customer recommendation.

How has the pandemic affected the need for continuous intelligence?

Petrie: One of the primary contributors to continuous intelligence -- or a common contributor -- is machine learning. As we all know with machine learning, there's a common problem with drifting data, drifting models, meaning that what you knew last month no longer applies to this week or next week. It's fair to say that some massive economic surprises in 2020 have accelerated data drift and model drift, so retailers need to reexamine all of their decisions and react very quickly to the very latest intelligence about customer shopping patterns, about local social distancing regulations, about all sorts of factors related to decisions that simply are very hard to predict because the world changes quickly. So I think that has accelerated and raised interest in continuous intelligence.

Historically, when most people think of streaming data, they think of entertainment apps like Netflix and Spotify, but when did continuous intelligence begin going beyond entertainment and start to permeate other sectors?

Petrie: I would say in the last two years, and much more this year, continuous intelligence has become a more common term and more common concept. You've got solutions from companies like Splunk that specialize in analyzing and helping operationalize actions on all types of machine data. You've got companies like Tibco that have specialized in streaming for years and are pioneers in the category that are helping organizations act on both machine data and application data. I think it's become more and more prevalent in the last year or two as organizations look to take advantage of new real-time technologies like Kafka, like Datastream, and operationalize those insights.

You just referenced this a bit, but are there technological platforms that enable streaming better than others or are most analytics stacks somewhat capable?

Petrie: Like a lot of technologies and disciplines, it's hard to buy continuous intelligence off the shelf. Often it involves an existing application data architecture or machine data architecture and starting to layer additional technologies on top of it. Sometimes it's a new commercial product, sometimes it's a new feature in a commercial product, and sometimes it's a new homegrown piece of code. But there are definitely some pretty innovative ideas out there.

I mentioned Splunk and Tibco, but there's also a company called Swim.ai, and what they do is combine digital twin technology and machine learning models to help different types of systems -- let's say IoT systems -- create real-time graphs that illustrate and track the relationships between different entities in a system and generate recommendations and real-time analysis about how those different pieces of the system relate to each other. It can help you manage a fleet of trucks, as an example, or manage flow in a telecom network.

As mentioned, the pandemic is forcing organizations to analyze their data more quickly the previously, but how deeply is continuous intelligence permeating the business world at this point?

Petrie: The first thing I'd say is that a lot of organizations might be using continuous intelligence and not have that label for it, but there's no question that organizations whose business models rely on e-commerce, on any type of customer data and generating insights from that, are starting to look at doing two things which really enable continuous intelligence. One is to look beyond their own data and look to external data for new findings and new opportunities, and then two, start figure out how to operationalize those real-time insights. I think there are a lot of companies doing that.

There are also a lot of companies that specialize in physical things that, especially in the wake of COVID, want to manage supply chains more efficiently so they're trying to make their things smarter and have lower latency decisions and insights about how to manage their supply chains in a more agile fashion. That may or may not drive continuous intelligence, but it starts to take them in that direction.

Is there anything else you'd like to add?

Petrie: Like a lot of terms, continuous intelligence gets used loosely by a lot of different vendors and different practitioners in different contexts, but the key takeaway is that enterprises are starting to strategically think about how to broaden the data that they analyze and how to operationalize the real-time insights they can derive from it. It's a pretty interesting trend that I think will drive a lot of adoption and a lot of data modernization in the COVID world in 2021.

How fast do you see adoption picking up momentum and could it be growing exponentially?

Petrie: Adoption is definitely growing, but I don't know whether it's exponential or not. It's definitely picking up, and I think it's a trend that's only going to accelerate.

Editor's note: This Q&A has been edited for clarity and conciseness.