Data warehousing for government

Data warehousing has been in the commercial sector for a long time now. The first data warehouses emerged in the late-1980s (they were called “atomic data bases” then). But in the early 1990s data warehousing took off commercially with the advent of ETL and OLAP processing. Soon, data warehousing blossomed into a full-fledged architecture known as the corporate information factory (the CIF). Data warehousing spread across the business world like wildfire.

Data warehousing started in the United States and eventually spread worldwide. Data warehouses are as common today in Malaysia as they are in Brazil, Australia, Europe and elsewhere.

Today, there is an urgent need for data warehousing in government circles. As the volumes of data grow large and the need for new and innovative information becomes manifest, it becomes apparent that the organization or agency needs a data warehouse. But surprisingly, data warehouses have been slow to be adopted in the government circles. Why? There are some fundamental reasons.

The most basic reason is that there is a significant difference in motivation for data warehousing in the commercial world and the governmental world. In the commercial world, the most fundamental motivations for data warehousing are to increase profit or increase market share protection. There are many other motivations for data warehousing in the commercial world, but these two motivations are the most basic and most visceral.

Government agencies, on the other hand, try to optimize their resources while building a data warehouse to the benefit of the constituency they reach. They are not concerned with reducing the size of their department due to budgetary reasons and political power. What are the visceral government motivations for a government data warehouse? The government motivations for a data warehouse are:

• The need for accuracy of data;
• The need for data at the lowest cost;
• The need for data at the fastest speed; and
• The need for integration of data.

Data warehousing addresses all of these needs. But, there are a lot of forces working against a data warehouse in government circles.

Working against the building of the data warehouse in government circles are attitudes such as:

• “Well that’s not how we did it 10 years ago.” And sure enough, 10 years ago you had significantly less data and significantly fewer demands for information on the government organization to produce information.
• “If I bring in a data warehouse, I am not going to need as many people.” That’s right, you can do things a lot more efficiently and inexpensively when you have a data warehouse.
• “I have to protect my data. No one else can look at it.” That’s just what the terrorists were planning on.
• “My SI vendor doesn’t understand the data warehouse; therefore there must be other ways to get the information.” All true. Get a new SI that does understand the data warehouse and information economics. (In fact, when was the last time you got a new SI? Is that healthy?).
• “My tour of duty is only two years. We won’t have much of a data warehouse built in that time, so it is going to hurt my chances of promotion to the next rank.” This is true for political appointees as well as military personnel.
• “A data warehouse costs so much money. Can’t I spend money this year to create some reports?” And next year spend some more money. And the year after that, even more, etc.
• “I only have a limited scope of assignment. A data warehouse goes well beyond my scope.” After this it becomes someone else’s problem.

And indeed there are hundreds of other reasons in government for not building a proper architecture. All of the excuses have a seed of truth. And all of them pale in comparison to the long-term need for efficient, accurate and cheap information in the government.

However, the basic truth and bottom line responsibility for the government remains that it is the steward of the public trust and must organize and maintain the information it accumulates in an efficient, and accessible and meaningful architecture for the future. 

Bill is universally recognized as the father of the data warehouse. He has more than 36 years of database technology management experience and data warehouse design expertise. He has published more than 40 books and 1,000 articles on data warehousing and data management, and his books have been translated into nine languages. He is known globally for his data warehouse development seminars and has been a keynote speaker for many major computing associations. Bill can be reached at 303-681-6772.

Editor's Note: More articles, resources and events are available in Bill's BeyeNETWORK Expert Channel. Be sure to visit today!