DataOps is a new tool for effective data use and improved data-driven decision-making. Organizations should prepare for these five DataOps challenges and learn how to overcome them.
As the amount of data has exploded in recent years, executives have faced monumental pressure to put all of it to good use. Yet they're often stymied in their efforts, turning to DataOps to effectively use data but encountering challenges along the way.
Ninety-seven percent of the organizations polled reported they were investing in data initiatives, but only 47.4% were competing on data and analytics, according to the NewVantage Partners "Data and AI Leadership Executive Survey 2022." Only 39.7% were managing data as an enterprise business asset, and a mere 26.5% said they had a data-driven organization.
In response to such poor showing, enterprise leaders are using the principles and practices within the DataOps discipline to help effectively use data to make decisions, drive insights and fuel automation and intelligence initiatives.
Out of 403 technical and business data professionals surveyed in the U.S. and Canada, 90% reported plans by their organization to make moderate to extensive investments in DataOps in the upcoming year, according to the August 2022 "The State of DataOps" report from Enterprise Strategy Group (ESG), a research firm and division of TechTarget.
"The heart and soul of DataOps is orchestration. Moving, processing and enriching data as it goes through a pipeline requires a complex workflow of tasks with numerous dependencies," said Neel Shapur, head of data advisory and architecture at professional services firm Genpact.
This discipline borrows concepts from the world of software development, such as agile development principles and the iterative and collaborative approach known as DevOps. It's meant to help data leaders within the enterprise securely deliver the right data to the right users at the right time by breaking down data siloes, adding automation and enforcing governance rules.
"DataOps delivers the ability to find, trust and understand data in a quick and reliable way," said JP Romero, data management practice lead at Kalypso, a consulting firm and IT service management company.
DataOps challenges
Organizations are adopting DataOps to overcome several challenges related to using their data, according to the ESG report.
Data leaders cited challenges around ensuring regulatory compliance, adhering to governance standards and gaining timely access to new data as top drivers for adopting DataOps.
Yet enterprise data leaders, researchers and executive advisors said organizations also face multiple challenges in successfully adopting and maturing the DataOps discipline.
Top DataOps challenges to implementing, using and scaling DataOps within the enterprise include the following:
1. A lack of clarity around what DataOps entails
The concept of DataOps has been around for nearly a decade now, but -- like its cousin, DevOps -- there's no single formula or all-encompassing guide to exactly what it entails. In fact, technology vendors typically have their own slightly unique take on what the discipline requires, said Jay Limburn, vice president of product management for data and AI at IBM.
Consequently, data leaders must identify the best practices that are emerging and the approaches that are beginning to become standardized to create a DataOps program that works for their enterprise, said Mike Hendrickson, vice president of tech & dev products at Skillsoft, a maker of learning management system software and content.
Data leaders should be agile, too, so they're ready to adapt as the discipline matures and new supporting technologies enter the market.
"DataOps is still early in the maturity cycle. Expect lots of changes and advancements both from a tool and process perspective," Shapur said.
2. An inability to find where, how to start
The volume of data is staggering. The world created or replicated 64.2 zettabytes of data in 2020, according to research firm IDC. Estimates project global data creation and replication will have a compound annual growth rate of 23% through 2025.
Although no organization has anywhere near that volume in their grasp, most still have more than they can manage. It can be difficult for them to know where to begin applying DataOps principles and how best to mature them, experts said.
"Define what success means to them" is where organizations should start, said Ramesh Vishwanathan, practice consulting director at IT service management company TEKsystems. It's the same advice he gives all his client companies.
Companies should identify areas where to pilot and practice a DataOps discipline before expanding it to involve all the data and data uses within the organization. In other words, take the minimum viable product approach and grow from there.
"Focus on a DataOps MVP that addresses a limited set of data use cases from the foundational elements to the value delivery," said Hector Rueda, data science technical manager at Kalypso. "Once that DataOps minimal viable product has proven its value, scale horizontally by expanding the scope."
Organizations should also find ways to measure the effectiveness of their DataOps programs.
"That will help you know if you're going in the right direction," Vishwanathan said.
3. A lack of data fundamentals
DataOps brings together people, processes and technology to orchestrate the effective, efficient and secure flow of data within an enterprise. To do that, organizations must have in place key components in all three areas.
More specifically, experts said organizations that are seeking to leverage DataOps should understand the following:
what data assets they have and the quality of that data;
how that data currently flows through the enterprise;
the technology components and talent they have to support all of those elements.
"DataOps needs a combination of technical investment, organizational restructuring and change management. There are technical, operational, and human and cultural barriers," Shapur said.
Yet, many organizations lack some or all of those elements. They also lack a data culture, as well as a data strategy.
The absence of those fundamentals can hold back attempts to successfully implement a DataOps discipline within the enterprise.
To counter that, data leaders should focus on building the foundational elements of a data program, so they have what they need to adopt DataOps.
Organizations need to put data and deployment of data facts front and center, rather than treat them as an "afterthought" within the software development cycle, said Dan Sutherland, senior director in the technology consulting practice at Protiviti.
Organizations need to be more attentive to all elements of the data lifecycle, including the design and development of the data pipeline, storage, data modeling and consumption patterns, Sutherland said.
Additionally, they should prioritize the following:
replace legacy data tech stacks with modern ones that provide complete visibility into the data pipeline;
invest in data literacy training across the enterprise; and
upskill their data teams, so they're prepared to work in this new environment.
Creating a solid data strategy that highlights the benefits of closing the gap between the current state and where the organization needs to use data to meet critical objectives is key.
"Identify senior executives who can support the program," Romero said. "Reach out to the people that will be positively impacted by the introduction of DataOps. Invest in a data literacy program that can foster a healthy data culture."
4. Lack of leadership buy-in
Another DataOps challenge organizations encounter is convincing leadership to back DataOps efforts. The more mature an organization's culture around data is, the easier this is to accomplish.
"It's easy for people who are using the data on a daily basis to say, 'We need this,' but leadership at times doesn't always see the need. They don't see the point in doing DataOps," said Romero.
Others echoed that comment, noting that executives in organizations without a mature data culture or gains from data-driven insights are often reluctant to support investments in DataOps.
It's easy for people who are using the data on a daily basis to say, 'We need this,' but leadership at times doesn't always see the need. They don't see the point in doing DataOps.
JP RomeroData management practice lead, Kalypso
Data leaders can overcome that lack of support by championing "the creation of data strategy and highlighting the benefits of closing the gap between where the data strategy is and where it needs to be," Romero said.
"Companies are slowly realizing that data must be considered a strategic asset," he said. "Those that connect their data efforts to their business's strategic imperatives will find it easier to unlock value from their data and fund their data programs."
5. Problems managing change
DataOps requires people to fundamentally change the way they work. This type of change isn't going to happen overnight, said Shapur.
"It's not easy to get staff to adopt new practices and agile working practices," he said. "Often, they lack skills and time to learn new skills. DataOps also requires a new software practices mindset."
As such, data leaders should fold change management principles into their DataOps plans to ensure they're able to successfully move people to a new way of working and thinking about data.
