As organizations grapple with ongoing data ecosystem complexity, the key question being asked is: how can organizations take back control of their data ecosystem and enable more people to better leverage data that matters to them? I believe it starts with data empowerment for all the key data stakeholders throughout the business, including both experts and generalists alike. Organizations need help in ensuring the right people are empowered to do their jobs more effectively and not get inundated with tasks that fall outside of their job descriptions. While this is true across the entire business from IT and operations teams to business analysts and developers, an area that continues to have a spotlight put on it is the data team, consisting of data engineers, data scientists, and ML engineers.
There are skills gaps virtually everywhere throughout data teams. The business needs more help than ever understanding what tools to integrate where and how to best leverage, manage, and maintain them. New technology and services are being made available across the data pipeline to new business units and people within new environments on what feels like a regular basis. With the time of the rock star data experts becoming increasingly valuable, bogging them down with mundane or less valuable tasks is costing companies millions in missed opportunities. A great example over the last several years can be seen on the data science side of the business, where data science teams are inundated with worrying about data integration as opposed to model building. We’ve heard the narrative of “we need more data scientists” for what feels like several years, but the fact of the matter is that the root of the problem occurs well before it hits the data science team. It is filling the role of data engineering. And it is a big reason why the #1 area where organizations are making the most significant data-centric investment over the next year is in data integration.
So where are data engineers asking for help? Teams that I’ve spoken with categorize their ongoing efforts into three buckets: building pipelines (ETL/ELT), building transformations, and making the right data available to the right team.
- Building and maintaining pipelines is hard, especially as organizations continue to embrace multiple environments encompassing on-premises and multiple clouds and look for ways to better unite data silos that are seemingly more distributed.
- Building transformations is increasingly challenging as more data is being requested that is not only growing and changing rapidly, but requires the merging of structured and unstructured data. And, oh, by the way, let’s not forget that most organizations have at least some sensitive data in their data pipeline that must be properly addressed to ensure compliance. Comprehensive data governance is becoming a must.
- Ensuring the right data is available to the right people is proving difficult as more end-users want to get their hands on more data, but lack of metadata creates challenges in understanding the data and therefore prevents organizations from being able to map that data back to ownership for proper business context.
Tools and platforms that can enable data teams, and specifically data engineers, to more effectively get their jobs done without being inundated with support and data requests are being prioritized. And fueling that adoption is the incorporation of self-service at the platform’s core, enabling automation in a way that helps data engineers take back control of their data ecosystem, while empowering end-users to gain more control themselves. Self-service is proving to be the fuel of data empowerment, and when implemented correctly, will go a long way in helping organizations achieve data-centric success.
