Analytics best practices include automation, cloud migration

Struggling to succeed

The goal of modern analytics is to derive insights from large datasets, noted Fern Halper, senior research director for advanced analytics at TDWI, who presented the findings of the vendor's survey during a webinar on June 12.

Common applications include supply chain management and optimization, personalized marketing, and fraud detection.

Harnessing data, however, is difficult. The volume of data that organizations now collect is growing exponentially. So is the complexity of data, which can be structured, unstructured and semi-structured.

As a result, organizations require complicated tools that incorporate such capabilities as machine learning, natural language processing and generative AI to develop data pipelines. Those pipelines import, integrate, organize and observe data to ensure its quality and ultimately help users analyze their data.

Also needed are self-service tools that make analytics use widespread within organizations rather than the domain of only data experts.

With all that's required to develop a strong analytics program, many organizations struggle to provide potential users with the training and tools to make analytics part of their workflows and expand their use of analytics beyond a dedicated team of data experts.

In the survey, less than 10% of organizations use advanced analytics capabilities, such as machine learning, NLP and large language models. More than 20% still use spreadsheets as their primary tool for analytics.

In addition, only 8% said they have been able to provide most business users in need of self-service analytics with the required training and tools.

"A large percentage of respondents say that they're planning to implement the technology, but they haven't done so," Halper said. "At the same time, organizations have had to deal with increased amounts of data and more complex data. So the need for more advanced analytics tools to … be more productive has increased. We have a conundrum."

Key challenges for those organizations struggling to move beyond spreadsheets and other rudimentary analytics capabilities include difficulty finding the right talent and providing proper training, a lack of funding for analytics, a dearth of tools able to meet the demands of modern analytics, and little support at the executive level, Halper continued.

"Organizational challenges lead the way," she said. "The top reason cited was -- not a surprise -- a lack of skills, which includes a lack of data literacy. That's a key roadblock in terms of analytics maturity. It blocks business users from using self-service tools."

It also blocks data experts from doing more sophisticated data science because they're instead building dashboards and reports and doing other simple tasks, Halper noted.

Challenges, however, are not limited to those organizations still stuck using spreadsheets. Even those that have taken steps to modernize often have difficulty implementing successful analytics operations.

Lack of talent -- particularly related to maintaining data and deriving insights -- is a problem for organizations attempting to modernize. So is the complexity of data and the algorithms needed to make it meaningful.

The cost of cloud computing poses still another challenge, particularly as organizations first move from on-premises data operations to cloud-based data operations.

"Especially when first starting out with the cloud, organizations may not be doing a good job monitoring cloud usage, and then the costs can get out of control," Halper said. "They haven't put the processes in place yet to monitor the usage."

There are, however, best practices that can enable organizations to overcome the many challenges modern analytics presents.

Best practices for analytics

Generative AI and LLMs may one day reduce the need for data literacy by enabling users to engage with data using true natural language.

But with the relationship between generative AI and analytics still in its nascent stage -- and with many organizations still using spreadsheets and struggling to implement intuitive tools -- data literacy training is critical to modern analytics success.

Data literacy is simply the ability to derive meaningful information from data. But deriving meaningful information from data is not simple. It takes education and training.

As a result, organizations in the survey that were successful with analytics overwhelmingly said data literacy is a priority compared with the unsuccessful organizations.

Data literacy is the foundation for self-service analytics. It's what enables business users to effectively analyze data and derive insights that lead to decisions and actions. Without data literacy, there can be no self-service analytics, and all data analysis is left to a centralized team of data experts.

When that is the case, inefficiency inevitably results.

End users are forced to submit requests for models, reports, dashboards and other data assets and then wait for the data experts to deliver the requested materials. It's a process that can take months depending on the complexity of the request and the number of requests submitted by others.

A large percentage of respondents say that they're planning to implement the technology, but they haven't done so. At the same time, organizations have had to deal with increased amounts of data and more complex data. So the need for more advanced analytics tools to … be more productive has increased. We have a conundrum.

"Data literacy training helps to democratize analytics," Halper said. "Successful companies recognize the importance of data literacy training, and they fund data literacy training. And then once business users are data literate, they can analyze their own data and release data analysts to do more sophisticated work."

Beyond recognizing the value data literacy, TDWI found that a best practice for successful analytics is prioritizing cloud migration. Most of the organizations with analytics success also prioritized using a cloud data platform.

Even more of those with analytics success also are automating many of their data management, data governance and analytics processes. Comparatively, among those with no success with analytics or only some success, half did not make process automation a priority in their organization.

"It's very difficult to [handle the complexity of modern analytics] manually," Halper said. "This is something organizations should pay attention to."

Ultimately it takes leadership to make data literacy a priority, make the move less onerous from on-premises to the cloud and invest in modern tools that help data workers by automating monotonous tasks.

Organizations that were successful overwhelmingly had an analytics leader, such as a chief data officer or chief analytics officer, at the executive level.

"Those in the successful group have a committed analytics leader," Halper said. "This leader is someone at the C-suite level who's a champion for analytics. They need to walk the walk, ensure that a strategy is in place, evangelize the need for analytics and fund analytics."

More best practices

In addition to data literacy, cloud migration, process automation and proper leadership, Halper noted that TDWI found other best practices common to those organizations most successful with analytics.

Collaboration and sharing are key, she noted. So are implementing an effective organizational structure and developing key performance indicators that are tied to business goals.

At the foundation of it all needs to be strong data and analytics governance frameworks that simultaneously govern data use to ensure security and compliance as well as encourage users to explore data to derive important insights.

"They're necessary to ensure that data is trustworthy," Halper said. "There's a limited number of chances you have to ensure that members of the organization want to use data for analysis. So in order for that to happen, you need to have good, quality data."

Eric Avidon is a senior news writer for TechTarget Editorial and a journalist with more than 25 years of experience. He covers analytics and data management.