Data silos and culture lead to data transformation challenges
Data continues to grow at a rapid rate, and users are faced with increasing complexity and challenges to make sense of it all in a way that helps improve business outcomes.
It's not as easy as it should be for many users to make full use of data for data analytics and business intelligence use cases, due to a number of data transformation challenges.
Data challenges arise not only in the form of data transformation problems, but also with broader strategic concerns about how data is collected and used.
Culture and data strategy within organizations are key causal factors of data transformation challenges, said Gartner analyst Mike Rollings.
"Making data available in various forms and to the right people at the right time has always been a challenge," Rollings said. "The bigger barrier to making data available is culture."
The path to overcoming data challenges is to create a culture of data and fully embrace the idea of being a data-driven enterprise, according to Rollings.
Rollings has been busy recently talking about the challenges of data analytics, including taking part in a session at the Gartner IT Symposium Expofrom Oct. 20-24 in Orlando, where he also detailed some of the findings from the Gartner CDO (Chief Data Officer) survey.
Among the key points in the study is that most organizations have not included data and analytics as part of documented corporate strategies.
Making data available in various forms and to the right people at the right time has always been a challenge.
Mike RollingsAnalyst, Gartner
"The primary challenge is that data and data insights are not a central part of business strategy," Rollings said.
Often, data and data analytics are actually just byproducts of other activities, rather than being the core focus of a formal data-driven architecture, he said. In Rollings' view, data and analytics should be considered assets that can be measured, managed and monetized.
"When we talk about measuring and monetizing, we're really saying, do you have an intentional process to even understand what you have," he said. "And do you have an intentional process to start to evaluate the opportunities that may exist with data, or with analysis that could fundamentally change the business model, customer experience and the way decisions are made."
Data transformation challenges
The struggle to make the data useful is a key challenge, said Hoshang Chenoy, senior manager of marketing analytics at San Francisco-based LiveRamp, an identity resolution software vendor.
Among other data transformation challenges is that many organizations still have siloed deployments, where data is collected and remains in isolated segments.
"In addition to having siloed data within an organization, I think the biggest challenge for enterprises to make their data ready for analytics are the attempts at pulling in data that has previously never been accessed, whether it's because the data exists in too many different formats or for privacy and security reasons," Chenoy said. "It can be a daunting task to start on a data management project but with the right tech, team and tools in place, enterprises should get started sooner rather than later."
How to address the challenges
With the data warehouse and data lake technologies, the early promise was making it easier to use data.
But despite technology advances, there's still a long way to go to solving data transformation challenges, said Ed Thompson, CTO of Matillion, a London-based data integration vendor that recently commissioned a survey on data integration problems.
The survey of 200 IT professionals found that 90% of organizations see making data available for insights as a barrier. The study also found a rapid rate of data growth of up to 100% a month at some organizations.
When an executive team starts to get good quality data, what typically comes back is a lot of questions that require more data. The continuous need to ask and answer questions is the cycle that is driving data demand.
"The more data that organizations have, the more insight that they can gain from it, the more they want, and the more they need," Thompson said.
