Getting started on building a real-time BI system: issues to consider

Real-time business intelligence (BI) has a definite cachet – given the choice, who wouldn’t want to work in real time? And it’s a capability that many organizations would like to have, for a simple reason: Business is speeding up and BI needs to keep pace.

According to a report on BI platforms issued late last year by Forrester Research Inc., BI is one of the few segments of the software industry to show much growth in recent years. For example, Forrester estimated that overall software sales were down 8% year over year in 2009, while sales of BI tools increased by 11%.

However, the Forrester Wave report stressed that BI isn’t a single product category – it includes a wide range of applications and infrastructure components. Conventional BI focuses on making historical data available for after-the-fact analysis by business users. A real-time BI system, on the other hand, aims to make analytics a part of tactical decision making on a daily basis.

Still, as Richard Hackathorn, a BI and data warehousing consultant at Bolder Technology Inc. in Boulder, Colo., points out, the term real-time business intelligence can be misleading. Low-latency delivery of BI data to users is usually a more realistic goal than actually making the information available in real time is, Hackathorn said.

Other industry analysts agreed, saying that operational BI is a better description of what most companies that have implemented tactical BI systems are currently doing.

“Most BI is not real time in the sense that the data is actually real time,” said Claudia Imhoff, president of consulting firm Intelligent Solutions Inc., also based in Boulder. She added that true real-time BI and analytics typically involve event- or stream-based technologies such as complex event processing (CEP) software.

In fact, the data used for “real-time” BI often can be anywhere from a few minutes to 24 hours old, or even older, Imhoff said. “The trick is to understand the business problem and determine if low-latency data is good enough,” she noted.

For example, BI data may not have to be fully up to date to enable users to predict when a product inventory will run out during a promotional campaign. Low-latency data delivery may also be suitable for analyzing and managing logistics issues, such as getting a product to a VIP customer at the optimal time.

Potential benefits of a real-time BI system
In addition to CEP tools, real-time BI can be enabled through change data capture technology and data virtualization software, among other means. According to Imhoff, the potential benefits include faster decisions based on more timely data; better customer satisfaction due to improved customer service; reduced “opportunity costs” in marketing and sales; and supply chain management savings through just-in-time deliveries and reduced inventory levels.

When it comes to constructing a real-time BI system, there are two primary approaches that companies can take, said Colin White, president of Ashland, Ore.-based consulting firm BI Research. One is to extend your existing data warehousing environment, and the other is to implement all-new technology.

“Either way, you must break the problem into three pieces: the time to get data, the analysis and the time to deliver that data to users to make a decision,” White said. Each of those areas has separate technical challenges, such as loading a data warehouse faster or implementing processes for pre-calculating analytic routines or speeding the data analysis process once information is in the data warehouse.

There are quite a few things that can be done in a traditional BI and data warehousing environment to speed up the information flow and help business users make decisions faster, White said. But like Hackathorn and Imhoff, he warned that there are limits.

Store and analyze in a real-time BI system – or analyze and store?
Most of the available techniques can move you toward data-delivery cycles of a few minutes after information is captured and stored, according to White – but if you’re looking for sub-second response times, it may not be feasible to bring data into the data warehouse first.

“If you want to get closer to real time,” White said, “you tend to look for what I call an analyze-and-store model, where you analyze the data as it goes through the system and then store the results.” Like Imhoff, he cited event- and stream-based analytics processes, in which data is captured, filtered and aggregated closer to the source system.

In short, there are many variations of real-time BI and ways to move toward it, depending on an organization’s specific requirements, IT infrastructure and existing BI investments. And it’s not all rocket science. However, Imhoff cautioned that real-time BI technology isn’t always the right solution to BI problems – and that it can create problems of its own.

“Even with real-time BI, organizations still have to worry about the integration of the data being analyzed,” Imhoff said. Furthermore, real-time BI tools can yield false positives or negatives that might lead to bad business decisions if not detected, “and the models used can rapidly become dated or out of sync with a changing business scenario,” she added. As a result, making sure that a real-time BI system is a good fit for your organization, and that your internal processes are set up to handle it, is critical to a successful deployment.

Alan R. Earls is a Boston-area freelance writer who focuses on business and technology.