Sergey Nivens - Fotolia
Business transformation project: Dealing with legacy data
Business transformation requires a solid data foundation. Expert Dan Morris offers a near-term fix for your data-quality problems and the basis for a long-term approach to better data.
Editor's note: This is the second in a series on the importance of data quality in business improvement or transformation. The first column in the series, "Resist 'garbage in, gospel out' doctrine -- defend data quality," can be found here.
Can CIOs deliver large-scale business improvement at companies where the data foundation is poor?
The obvious answer to this question is no, but as with most issues in IT, fixing the problem is complicated.
The legacy systems that CIOs must contend with today as they take on a business transformation project were once considered state of the art. And the legacy data management practices CIOs were once urged to follow were thought to be best practices.
Time has proven that many of these best practices may not really have been more than expedient practice. The proof is in the sheer numbers of legacy applications and related databases at most companies and the impediment they've become to delivering large-scale business improvement. Poor data is a problem at almost every company. The result is that business users, with good reason, have little faith in the quality, completeness or currency of the data in their reports.
How do we get from this state of distrust in company data to launching a business transformation project that is likely to produce an outcome that business managers and staff can trust?
For visionary companies, the first step in taking on a business transformation project is creating a flexible technology environment that is synced up with the business operations at all levels of detail.
There are technologies that can reduce complexity in the business operation's work and workflow. We have technologies to streamline the business operation to improve efficiency and add in performance monitoring to check on efficiency and error rates. We can apply advanced business operating concepts, roll in Six Sigma performance checking, create new ways of interacting with customers and more. We can automate much of the work with modern business process management software, robotic process automation and AI technologies.
But to really utilize IT's ability to help transform business models, IT departments must look at moving to new technology architectures and not simply cobbling more technologies into the current amorphous enterprise architecture. This is complex engineering and requires business transformation experience, creativity and a willingness to innovate, but it can be done -- and it must be done. This IT transformation is the foundation of a successful business transformation project.
A process for fixing legacy data
The best business redesign, however, is doomed to fail if the IT infrastructure cannot support rapid change. And both the new business design and the new flexible IT infrastructure will be restricted -- or worse, fail -- if the data is of poor quality. At a minimum, people will not know if the data sources in published reports or the data used to train the algorithms are wrong; or if the other external information is in error; or if the data is incomplete or so old that it really has little value.
So, what can you do?
It will cost tens of millions of dollars and years to clean legacy data using traditional technical approaches. Practically, these approaches are not feasible. But, there is an interim way.
If you look at processes and then identify all the data in the process you are redesigning, you will need to determine what legacy data will be needed, when and where. The next step is to identify "sources of record" or sources of clean data. The data that will be needed to support uses can then be pulled on a given cycle and be combined to form transient databases. This precludes true real-time access, but it does allow near-real-time access. The new applications will access these temporary databases to create their reports. Responses from the business operation will be held and then released on a given cycle, to be sent back to the source legacy application for updating. This should be considered a temporary fix. But it will acquaint the data sciences people with the issues, needs and support the ongoing business operation requires while other permanent steps are defined and taken.
This next step should be a plan to clean critical data and eliminate everything that is, in fact, not critical. This eliminated data should be archived. For all the critical information, the data architects should be able to define what needs to be done to clean the data using one of the many data cleansing tools available; according to Gartner, there are numerous good products that can help improve the quality of legacy data. These include products from Informatica, SAS, SAP, IBM, Oracle, Syscord and many more.
Long-term vision
After a proof-of-concept project, it will be possible to create an approach that works in your company and give you the basis for a multiyear roadmap and investment plan for legacy data cleansing. This is a bonified business transformation project, and it can now be managed against formal targets and timelines.
In this way, the new business operations and their computer applications can evolve along a specific path that includes data cleansing using advanced data management practices. This will not be a few months' effort and it will not be simple. But, to avoid making decisions on poor data, increasing the probability of processing errors and presenting erroneous data to customers, this may merit a high-priority status for both the business and IT. Of course, like any business transformation project, it will require a strong executive commitment to change business and IT operations and a fundamental understanding of how data management affects customer service.
