CIOs face obstacles when scaling generative AI

Investing in data

Among those issues are data availability and data quality, which are foundational to generative AI rollouts. The performance and reliability of such systems depends on the accuracy and relevance of the data. Yet enterprises might be tempted to neglect the data step and rush into deployment.

"There's a ton of excitement around deploying AI in general, and GenAI specifically," said Anish Athalye, CTO at Cleanlab, a San Franciso-based data curation and quality tool provider. "But we aren't yet at a point where it's common knowledge that data quality is a critical piece of the puzzle."

Athalye, whose company participated in the CIO Symposium's Innovation Showcase, said organizations at the forefront of AI understand the importance of data quality. The degree of data savvy below that tier varies, however.

"They may or may not be at the point where they realize that this is not an optional thing to do," he added.

Once organizations commit to a data initiative, the problem becomes project scope. Baig said many organizations take a top-down approach that identifies GenAI use cases and seeks the data required to support them. The result is a massive project unlikely to be completed within a reasonable amount of time, he noted.

Instead, Baig advised organizations to focus on a handful of data domains that can fuel multiple, high-priority use cases.

"That usually ends up being three or four domains that you can actually get started on," he said.

Tracking regulatory developments

CIOs, whether they're delving into GenAI or other tech initiatives, will also need to keep tabs on data privacy and security regulations and regulatory compliance duties. Stuart Madnick, co-founder of the Cybersecurity at MIT Sloan research program, said his group is studying more than 170 new cybersecurity regulations that affect IT.

"These regulations and legislation are coming from everywhere -- from the White House, from Congress, from almost every three-letter agency you can throw at it," said Madnick, who spoke on cyber resilience at the MIT event.

In addition to the U.S. federal sector and state governments, he noted that the EU, India and China contribute to the regulatory landscape internationally. Governments pursue AI regulation to address consumer protection and intellectual property rights, among other concerns.

Gayatri Shenai, a senior partner at McKinsey, said IT leaders must consider security, privacy and regulatory compliance when selecting GenAI partners. Those factors should be added to traditional partner evaluation criteria such as financial viability, according to Shenai, who moderated a panel discussion on managing multivendor partnerships at the CIO Symposium.

She framed the question for IT leaders: "How do we establish an awareness of compliance requirements and modify the [partner selection] guardrails we might have had?"

Narrowing the use case list

Enterprises have brainstormed scores of GenAI use cases since the technology went mainstream toward the end of 2022. The task now is focusing on the ones most likely to yield tangible results, industry executives said.

There are a number of issues to solve before you go from pilot to scale.

Enterprises might launch a multitude of generative AI projects, but few of those produce bottom-line benefits, Baig said. McKinsey's global AI survey found that only 15% of companies are seeing earnings improvements from generative AI, he noted.

"One of the most important roles a CIO can play is to get the organization to focus on initiatives that will drive real business value, which is solving a critical business problem that is technologically feasible," Baig said.

A solid business case becomes more important as generative AI applications scale -- and expenses begin to add up. Some of those costs might not be immediately evident, Baig noted. The low upfront costs of GenAI belie the costs of running and maintaining a system over time, he said.

Beyond technology, change management costs could grow to three times the cost of a generative AI deployment, Baig added.

Orchestrating the deployment

IT leaders must also assemble a multilayered technology stack to deploy generative AI.

"Even a simple generative AI solution requires about 20 to 30 elements that you have to put together," Baig said.

Those elements include a user interface, data enrichment capabilities, security and access control, and an API gateway that links to the foundation models. Automating technical workflows, such as model testing and validation, is also important for fully realizing generative AI's benefits, Baig said.

Another complication is integrating GenAI tools with enterprise IT environments, including legacy systems. Conference attendee Darlene Williams, senior vice president and CIO at Rocket Software, said a wealth of data exists on corporate mainframes that could be used to train AI models.

"I think mainframes absolutely drive AI," she said, citing generative AI and predictive AI as examples.

Rocket Software, an IT modernization software company based in Waltham, Mass., earlier this month completed its acquisition of OpenText's mainframe modernization business.

Enterprises embarking on complex GenAI deployment and integration projects will need to coordinate multiple partners as well as myriad technologies. This calls for greater collaboration among more stakeholders, McKinsey's Shenai said. Those parties might include product owners; an AI safety, trust and responsibility group; and an IT sourcing organization, as well as project and program managers, she noted.

"Those are pieces that people are not talking enough about," Shenai said.

John Moore is a writer for TechTarget Editorial covering the CIO role, economic trends and the IT services industry.