Business intelligence needs boost from decision intelligence
As the volume and complexity of data grows, organizations need AI and ML capabilities to surface insights and augment the capabilities of overmatched BI tools.
Business intelligence now needs decision intelligence.
BI is the use of data to inform decisions. Decision intelligence, meanwhile, is the use of augmented analytics and machine learning to automatically surface insights that lead to decisions and action.
For decades, BI has been used to build reports and dashboards that have let organizations make smarter decisions than they would have using only the instincts developed through experience.
BI platforms -- from Cognos and BusinessObjects a couple of decades ago to the modern Microsoft Power BI, Qlik and Tableau -- have been critical means of helping organizations compete with their peers by using data to inform decisions and even gain advantages.
They help data teams build models that predict future outcomes and foster self-service exploration and analysis by business users, which lets them make decisions in the moment that spur their organization's growth.
Limitations of BI
But BI has limitations, according to Wayne Eckerson, founder and principal consultant of Eckerson Group, who spoke during a recent webinar hosted by analytics vendor Sisu.
"The promise of BI was to turn data into insights and action, and it gets us part of the way there," he said. "But we've discovered over the years that it doesn't get us all the way there. It doesn't go that last mile to go from data to insights and action. There's nothing wrong with BI, but it has built-in limitations."
Sisu's Joel McKelvey (left) and Eckerson Group's Wayne Eckerson discuss decision intelligence during a recent webinar hosted by Sisu.
Eckerson noted that BI's limitations include the following:
It's not actionable because it's based on historical data.
It's too generalized, showing summarized trends and patterns that can miss subtle nuances that drive potential changes.
It's too manual, requiring humans to sift through copious data to find relevant information.
BI's constraints, meanwhile, are being exposed by the expontential growth in the amount of data organizations now collect and the increasing complexity of that data as it flows in from an increasing number of sources.
The result of BI's limitations is a decision-making bottleneck. The data is too much for humans to manage, and its complexity is beyond the scope of self-service users.
Just like how data teams building reports and dashboards was a slow process before the rise of self-service analytics -- with data consumers waiting weeks or even months for data teams to complete a given project -- data teams are again being overwhelmed by questions about data, and projects are again getting stalled.
Addressing the bottleneck
A few weeks or a couple of months to produce a report or dashboard may have been good enough a decade or two ago. Now that more of an organization's peers are as data driven as they are, it isn't. Additionally, fast-changing economic conditions due to worldwide events such as the pandemic and the war in Ukraine require organizations to act and react quickly.
"Solving the bottleneck -- other than data quality -- is the biggest issue facing the analytics industry," Eckerson said. "Business users just can't get the insights they need to take action."
And neither hiring more data analysts and data scientists nor developing better self-service analytics tools are the way to loosen the bottleneck, he continued.
As data volume and complexity increase, there will never be enough data workers to keep up nor enough money to pay them. Adding more self-service users and technology requires expensive data literacy training and stringent data governance to control potential chaos.
Instead, data teams need technology that makes them more efficient. That technology is decision intelligence.
"Decision intelligence platforms are like hiring an army of data analysts without spending any extra money, other than on a software license," Eckerson said.
Joel McKelvey, vice president of product marketing at Sisu -- a vendor that, like Pyramid Analytics and Tellius, specializes in decision intelligence tools -- likewise noted that BI needs decision intelligence capabilities to improve efficiency and meet organizations' modern needs.
"BI has been incredibly successful," he said. "But the data has grown -- and the complexity of that data has grown -- beyond our ability to serve it to just anybody in the company. I don't think BI is bad, but the implementation of a reporting and dashboarding tool has been outgrown. What we need now is a tool that automates much of what BI does."
The need for decision intelligence
Decision intelligence essentially uses AI and machine learning to monitor data.
Every organization has a set of business metrics that are most important to its success. Their data teams can program a decision intelligence platform to perform around-the-clock surveillance on those metrics and the data that drives them.
Decision intelligence platforms are like hiring an army of data analysts without spending any extra money, other than on a software license.
Wayne EckersonFounder and principal consultant, Eckerson Group
Any time there's a change in those metrics, the decision intelligence tool automatically alerts key stakeholders.
But they don't simply monitor what is happening. Because they can sift through millions of data point combinations in seconds, they can reveal why metrics are changing and suggest the most likely causes, saving analysts time otherwise spent performing root cause analysis.
The result is data workers becoming 10 to 100 times more productive than with a BI tool alone, according to Eckerson.
"Decision intelligence is about empowering data analysts and data scientists with a powerful engine that will rifle through millions of records in sub-second time and surface relevant issues that you need to deal with," he said. "You could never analyze your data that way as even an army of analysts, but these tools do that. They lift to the surface things that are worth looking at by the analysts that you do have."
That said, decision intelligence tools should not replace BI platforms, according to McKelvey.
Instead, decision intelligence should complement BI by surfacing insights that otherwise take months to discover and offering suggested subsequent actions. Despite its automation capabilities, decision intelligence should only be used to automate certain basic and repeatable actions resulting from automatically surfaced insights.
"Decision intelligence is about getting to a decision," McKelvey said. "It's not correct for us to assume decisions will be automated through this process. It really is about getting people to the point where decisions are data-driven and clear. It takes machine-scale data and winnows it down to a point where humans can take action in a meaningful way."
They understand certain distinctions that machines may not. Therefore, while decision intelligence can automate certain straightforward processes and decisions -- "if X happens, then always do Y" -- more nuanced decisions require some human interpretation, according to McKelvey.
"We use decision intelligence to augment the human, not replace them," he said.
Humans also are better than machines at distinguishing if something seems a little off.
While machines can sift through data and surface insights exponentially faster than humans, they aren't necessarily better at recognizing when an insight might be based on bad data and, therefore, not correct. An action taken based on bad data can prove disastrous.
"Decision intelligence does not solve data wrangling and data preparation issues," Eckerson said. "You still need good data. It's garbage in, garbage out -- you want to make sure you're delivering high quality data, because the better-quality data you have, the better-quality results you're going to get."
Ultimately, the goal of decision intelligence is to detect what happened, analyze why it happened, forecast what will happen as a result, prescribe potential responses and act on that prescription.
"We're not quite there yet," Eckerson said. "But I think in three to five years, this is where the vast majority of the technology will be. They're going to evolve very quickly in the next several years. The really cool features are coming down the road, such as the ability to do forecasting, prescription and even taking action, if needed."
