Big data analytics and business intelligence: A comparison

BI vs. big data analytics

Starting in the 1990s, the early years of BI marked a shift from static reports to a systematic data analysis practice that delivered aggregated data and KPIs to business executives. BI systems typically were, and still are, built on top of data warehouses that store large volumes of historical data optimized for analytical queries -- essentially providing a structured model of the business.

To enable this new mode of analysis, user-friendly BI tools emerged in the form of BI dashboards and interactive analytics applications. They led to the development of self-service BI environments, which let business users run queries and analyze data themselves instead of relying on skilled BI professionals. Today, BI encompasses a wide array of technologies and practices for collecting, storing, analyzing and visualizing data from diverse sources. Its aim is to provide actionable insights and support decision-making across all levels of an organization.

The subsequent emergence of big data analytics further expanded the possibilities for generating useful insights. While the definition of big data often focuses on the scale and complexity of the data involved, it's about more than just those things. Big data typically is characterized by the following three key elements, as originally noted in 2001 by Doug Laney, at the time an analyst at Meta Group Inc.:

• Volume. Big data often involves data sets that are far larger than traditional data processing tools can handle.
• Velocity. The data often is generated and updated in real time or near real time.
• Variety. Big data encompasses diverse data types, including structured, unstructured and semistructured data.

Known as the 3 V's, these characteristics have since been extended to include others -- most commonly veracity and value for a total of 5 V's. Developed in response to these data demands, big data analytics aims to extract insights that conventional BI methods either can't deliver at scale or might not discover at all. Such insights include hidden patterns in data sets, correlations between data elements and data trends over time.

Key differences between BI and big data analytics

While both BI and big data analytics look for new insights in data, they differ significantly in terms of data architecture, processing methods and analytical focus, as explained below.

Data architecture and data processing

Business intelligence applications primarily work with structured data sets from internal sources, such as transactional systems, SQL databases and even spreadsheets. These data sets typically are organized in rows and columns for easier processing. Query languages such as SQL or Multidimensional Expressions, commonly known as MDX, are used to produce summaries, reports and data visualizations. As mentioned previously, the data is usually stored in a data warehouse for querying and analysis.

In contrast, big data analytics often handles vast amounts of data in diverse formats. This data also originates from varied sources, such as sensors, emails, images, external databases and social media. Some of these data sets require advanced processing techniques to extract meaningful insights. Others involve processing at such a scale that it requires distributing data analysis jobs across multiple servers and then bringing the results back together to be consolidated and delivered to the user.

This level of processing is done with big data technologies such as Hadoop and Spark; machine learning algorithms; and scripting languages such as Python and R, among other tools. The data is commonly stored in raw form in a data lake, where it can be analyzed as is or filtered and prepared as needed for specific analytics applications.

Another difference is that while BI typically deals with periodic data updates -- daily, weekly or monthly, for example -- to summarize historical performance, big data analytics systems are designed to process data faster, including in real time. This capability lets organizations quickly respond to changes in the market, customer behavior and operational conditions.

Analytical focus

BI mainly focuses on descriptive analytics and diagnostic analytics, looking at what happened previously and why it happened to help businesses make informed decisions about current strategies. It can also be employed in a similar way to monitor KPIs and identify operational improvements.

Big data analytics can include these kinds of analysis, too, albeit usually at greater scale. However, it most often goes beyond them to predictive analytics, which aims to anticipate future scenarios and trends. Prescriptive analytics that offers guidance on how to achieve a desired business outcome is also a standard practice with big data.

BI is regularly used by corporate executives and business managers for tactical and strategic decision-making. Big data analytics, on the other hand, is most often done by data scientists for more forward-looking purposes, such as predicting market trends, optimizing supply chains, enhancing the customer experience through personalization and driving innovation in product development.

How do BI and big data analytics benefit organizations?

While both business intelligence and big data analytics aim to improve decision-making, they serve different purposes and, as we've seen, use different technical methods. As a result, different business benefits emerge from each technique. Let's look separately at their potential benefits.

Business intelligence: Structured approach to data analysis

Think of BI as a model-driven approach to data analysis. BI applications are built on predefined business models that capture a shared understanding of how the organization operates or should operate. Also, BI is often process-aligned. The models are structured to mirror existing business workflows and organized according to strategic, tactical and operational levels.

In addition, the insights from BI relate to standardized, consistent KPIs and other metrics that can be shared and monitored across an organization.

There's a great advantage to this approach: It directly maps to the organization's strategic objectives. This mapping enables consistent tracking of key business initiatives and also makes corporate governance and compliance processes easier to manage.

Another advantage of this structured data analysis approach is that BI can identify bottlenecks and inefficiencies in business processes and measure their performance against established benchmarks. Any business that wants to support continuous improvement or quality management in its operations is likely to use BI as a tool to help drive both tactical and strategic decisions.

Big data analytics: Exploratory data analysis approach

In contrast to the structured nature of BI, big data analytics takes a more exploratory approach. Rather than being steered by existing models of a business, big data analytics applications typically are driven by the data as it's generated and collected by an organization. For example, they can discover patterns in raw data that might reflect unexpected changes in the market or a company's operating environment.

Data scientists use techniques such as hypothesis testing and predictive modeling to identify, analyze and test these data patterns. This increases the ability of businesses to adapt their strategic and tactical initiatives. The business insights that are generated support the development of new processes and help identify market risks and opportunities.

The comparison table below summarizes the different advantages of BI and big data analytics for businesses:

How BI and big data can be integrated

The most effective organizations employ both analytics approaches. They use BI to help execute their current business strategy and optimize their operational tactics. But they complement it with big data analytics to enable strategic evolution, adapt to new developments and operationalize business innovation.

It's also possible to integrate big data into BI processes. For example, big data technologies can be used to extract structure from unstructured data sources in such a way that BI tools are able to work with the data. Or big data analytics can deliver predictions, such as future customer lifetime value scores, as tables that BI tools can integrate into a customer service dashboard.

Similarly, data scientists might integrate BI into big data analytics workflows, especially if they need to use well-structured data in their analysis work. For example, a data warehouse might contain customer or patient data carefully structured for regulatory compliance in industries such as financial services and healthcare. A data scientist could use this cleansed and conformed data set, instead of the unprocessed raw data, to ensure the generated insights are actionable within the applicable regulatory framework. There's no point in discovering a new analytical insight if the company can't legally take advantage of it.

A new platform architecture, the data lakehouse, has emerged to support such integrations. As the name suggests, it combines key features of data lakes and data warehouses, providing a single platform for both BI and big data analytics applications.

Examples of BI and big data analytics applications

The following table gives some examples of different application scenarios for both business intelligence and big data analytics to show how they contrast with -- and complement -- each other:

Organizations that effectively integrate both approaches are well-placed to evolve and adapt their business operations, and to gain a competitive edge over rival companies that don't do so. As the volume, velocity and variety of data continue to grow, and as global business environments become more and more complex, the technical distinction between BI and big data analytics might start to blur. Even if that happens, however, their fundamental purposes likely will remain distinct.

Successful businesses won't choose between BI and big data analytics. Instead, they'll take advantage of the strengths of each approach to create a comprehensive data analytics strategy that serves both current business needs and requirements for future innovation.

Donald Farmer is a data strategist with 30-plus years of experience, including as a product team leader at Microsoft and Qlik. He advises global clients on data, analytics, AI and innovation strategy, with expertise spanning from tech giants to startups.