Active data warehousing explained and examined

Business intelligence (BI) is becoming an important part of more organizations' day-to-day operations, and active data warehousing is a prime enabling technology.

As with many good business technologies, the concept of active data warehousing was born from user requirements rather than from vendors' technological ability. With active data warehousing, we aren't talking about supporting executives' strategic decision-making -- we are talking about operational decision support. For example, shipping companies like DHL use huge fleets of trucks to move millions of packages around. Every day, operational managers are making thousands of nitty-gritty decisions that affect the bottom line: "Do we need three trucks for this run?" "With two drivers out sick, do I need to bring in extra help?" Traditional data warehousing doesn't help with this, but active data warehousing does.

Most people are familiar with the idea of a data warehouse -- a system that pulls together data from disparate sources in order to run analytical queries against huge data sets amassed over long periods. Such systems bring immense business value by facilitating the analytical overview that steers the direction of a company. For instance, analysis can reveal changes in customer profiles and drive the decision to redevelop the advertising strategy. These benefits are at a high level, however, and the day-to-day operations staff of an organization often do not derive much benefit from a traditional data warehouse. What makes active data warehousing different is its ability to deliver help at the operational level.

I should also note that terminology is somewhat fluid in this relatively new field. "Active data warehousing" and "third-generation BI" are often used to refer to similar kinds of technology implementations, though elements of "business activity monitoring" are in there too. What's really important, however, is the potential for your business enabled by this kind of system -- regardless of the name.

What can active data warehousing do?

Consider this scenario: An employee at a bank's help desk is dealing with an angry customer. The customer has allowed her checking account to become mildly overdrawn and is faced with a $25 service charge: "It's not fair, it was only a few dollars over a few days, I'm a good customer," she might say. The help desk employee has to decide on the spot whether to apply the charge or to waive it. Is this really a "good" customer? (After all, how many callers will claim to be a "bad" customer?)

A wise decision here will be either to claim the $25 from a costly, high-maintenance "bad" customer or waive the $25 to placate and keep a "good" profitable customer. A foolish decision will, of course, do the reverse and may lose the profitable customer.

A traditional data warehouse doesn't begin to address this situation; instead, the help desk employees are left to their own devices. An active data warehouse would be involved from the start, identifying profitable/desirable customers and would either automatically make the decision to apply or withdraw the charge or provide strong advice to the help desk employee.

Of course, this technology is very broadly applicable. There are other examples of active data warehousing in place today. It isn't just about CRM. It can be used, for example, to charge different customers different rates for the same service, which brings a whole new meaning to "a personalized bill."

The largest insurer in the U.K., Norwich-based Norwich Union, has developed a "pay as you drive" insurance policy with rates starting at two cents a mile. Cars are fitted with a GPS device that allows Norwich Union to track the car and construct the policy around each individual motorist. Each day, every single trip made by every customer is analyzed in an active data warehouse system from Miamisburg, Ohio-based Teradata Corp. The information gleaned from this system is passed on to the billing system and used to produce individualized customer premiums. Customers then receive a monthly bill that is based not just on the number of miles covered but also on the time of travel, the type of road used, and so on.

Active data warehousing's technological implications

The types of operational advantage that active data warehousing offers, like the insurance or help desk examples above, impose a very different workload on a data warehouse. In the previous model of warehousing, a small number of very complex queries were run against the warehouse each day. With an active data warehouse, this number can reach millions of queries per day, which puts the workload firmly into the category normally associated with online transaction processing (OLTP) systems.

Some vendors, such as Teradata, have already developed warehousing products with architectures capable of handling this workload. The new data warehouse appliances that have appeared over the last several years are often very well adapted to this as well. Manufacturers come to mind such as U.K.-based Kognitio Ltd.; Aliso Viejo, Calif.-based Datallegro Inc.; and Framingham, Mass.-based Netezza Corp.

The concept seems to hearken back to traditional database goals, agrees David Norfolk, senior analyst for development with Towcester, U.K.-based Bloor Research.

"In the extreme, [active data warehousing] represents a return to the original database concept," Norfolk said. "All the information in the organization is available for decision and business transaction support in near real time. This represents a very different data warehouse load to that expected by most data warehouses today, [where] a day's processing may be millions of very specific, fairly simple, queries instead of hundreds of complex analytical queries."

"However, it is a load with many similarities to enterprise OLTP loads," he continued. "So, while only the most advanced data warehouses will cope with active data warehousing, perhaps the sort of general-purpose databases that can already cope with small data warehouse queries and OLTP against the same database -- or even appliances that can present different virtualized views into a common data store -- will find that active data warehousing brings them a real opportunity."

This may change the competitive landscape for enterprise databases and data warehouse products, Norfolk added.

"Perhaps the general-purpose DBMS will come to dominate the marketplace, especially for the smaller enterprise, and this will take the bread and butter away from the data warehouse specialists," he said.

Active data warehousing's competitive advantages

The whole active data warehousing field has been made possible largely by the massive increase in computing power -- processing speeds, memory, etc. -- that is now available. Active data warehousing is a fascinating technology in its own right, but it is of no value to a business if that business has no need for what it offers. On the other hand, all businesses crave competitive advantages, and active data warehousing certainly has the potential to bring many advantages. It's hard to imagine any business making thousands or millions of decisions at the customer level unless it implements some form of active data warehousing to enable this kind of operational decision-making.

About the author

Dr. Mark Whitehorn specializes in the areas of data analysis, data modeling, data warehousing and business intelligence (BI). Based in the U.K., he works as a consultant for a number of national and international companies, designing databases and BI systems. In addition to his consultancy practice, he is a well-recognized commentator on the computer world, publishing about 150,000 words a year, which appear in the form of articles, in publications such as PCW and Server Management Magazine, white papers and books. He has written nine books on database and BI technology. The first one "Inside Relational Databases" (1997) is now in its third edition and has been translated into three other languages. The most recent is about MDX (a language for manipulating multi-dimensional data structures) and was co-written with the original architect of the language – Mosha Pasumansky. Mark has also worked as an associate with QA-IQ since 2000. He developed the company's database analysis and design course as well as its data warehousing course.