Ally's generative AI strategy eyes multiple LLMs, AI agents

Managing multiple LLMs

The Ally.ai platform provides a vendor-agnostic environment that will, over time, provide access to multiple large language models (LLMs), said Sathish Muthukrishnan, chief information, data and digital officer at Ally Financial. The platform's initial LLM tool is Microsoft's Azure OpenAI Service, which provides access to models such as GPT-3.5 Turbo and GPT-4.

But the company is now vetting Amazon Bedrock, a machine learning platform for building GenAI applications. The governance and approval process for assuring Bedrock is secure enough to put into production will take about three to four months, Muthukrishnan said.

The multimodel approach will let users query the model of their choosing, float an inquiry across several models and obtain a consolidated and summarized result, or receive answers from each model and determine the best response, Muthukrishnan explained.

Organizations that have worked with GenAI for a while and at scale often create private "modeling gardens," which expose models to internal users through an API, said Arun Chandrasekaran, an analyst with Gartner. This "composable approach" provides the flexibility to work with multiple models, whether they're large or small as well as proprietary or open source, he noted.

"Composability is a theme we are increasingly seeing with early adopters," Chandrasekaran said. "They want to have the ability to swap models in the future and reduce the lock-in to any specific vendor."

John Kain, head of financial services market development at AWS, also pointed to Ally Financial's GenAI platform being designed to accommodate more than one model as indicative of a wider trend.

"There is not one model to rule them all for customers to be successful," he said. "They need to fit the right model to the right application. I think that is where the industry is going, broadly."

Security is another factor with private gardens. Ally.ai provides a buffer between the company's users and third-party models.

"We have the ability to control the input and the output with the LLMs," Muthukrishnan said.

Ally.ai privately hosts LLMs and GenAI applications while also handling request processing. Users are authorized and authenticated through an API gateway. The platform ensures personal information isn't leaving Ally Financial's network through a user inquiry and lets the bank assess LLM responses, Muthukrishnan said.

"I can review and inspect the data to make sure it doesn't deviate from the task it was asked to do," he said.

Exploring GenAI agents

Autonomous AI agents could emerge as another aspect of Ally Financial's GenAI initiative. Muthukrishnan said the company plans to experiment with Bedrock's ability to create such agents. Autonomous agents are designed to take action rather than provide information as they automate tasks. Industry executives identified autonomous agents as among the top GenAI trends expected to unfold in 2024.

Muthukrishnan described autonomous agents as the "next level" for GenAI as the technology shifts from LLMs to LAMs, or large action models. An LLM collects, synthesizes and summarizes information, he said. But an LAM "acts on the information, so the agents built on the LAMs can start to do tasks like a human," he added.

At Ally Financial, an autonomous agent might eventually serve as a "product owner assistant" on an Agile software development team, Muthukrishnan said. The product owner on such a team oversees a development project, monitors progress and manages backlog items. An autonomous assistant would track project execution, making sure Agile teams are on schedule and deliver the code they signed up to contribute, Muthukrishnan noted.

Such schedule checks would traditionally take place in a daily Scrum, in which developers report on the status of their tasks. But a GenAI agent could potentially handle those updates, reducing the number of meetings and letting developers focus instead on complex problem-solving, Muthukrishnan said.

Boosting developer productivity

While automated agents have yet to materialize, Ally Financial currently experiments with GenAI to generate code. Muthukrishnan cited a 25% to 35% productivity gain for the bank's developers using Azure OpenAI Service. He said he expects an equivalent boost from Bedrock once it's in production.

Chandrasekaran said 25% to 35% developer productivity gains are somewhat above the median of what Gartner has seen among enterprises. Increases of that magnitude generally depend on organizations having a high degree of automation across the software development life cycle (SDLC) in addition to GenAI, he said.

"Not all organizations have that level of automation baked into the [SDLC]," he said.

In Ally Financial's case, developer productivity stems from its tech operating model and its use of cloud, test automation, release pipelines and GenAI, Muthukrishnan said. That combination means the bank's technology team, funded for 100% of its development capacity, can deliver up to 120% of that capacity, he noted.

As a result, the team can handle growing development demands with the same level of human resources and budget allocation, he added.

Cultivating GenAI use cases

OpenAI launched ChatGPT in November 2022, quickly increasing adoption of generative AI tools. But Ally Financial banned employees from using GenAI when it first emerged until the bank could establish governance controls.

We believe this is not a technology-only transformation. This is a company-wide transformation.

Today, Ally Financial's approach is to selectively open the Ally.ai platform to employees, Muthukrishnan said. Employees with access and an idea for a use case are given a month to experiment with the technology. At that point, they present a whitepaper, which details the use case, the anticipated investment and the expected business impact.

A working group representing the bank's technology and business teams evaluates the use cases and prioritizes them for deployment, Muthukrishnan said. The technology members consider such factors as the amount of time it will take to execute the use case, while the business members assess the importance of the use case to customers.

Other enterprises have also taken a bottom-up approach to generating GenAI use cases. Walmart holds listening sessions to get employee feedback on the retailer's My Assistant tool. It learns how employees working at corporate facilities use the GenAI tool and gauges the potential for scaling the various use cases that surface.

Training for AI transformation

Ally Financial provides guidance and training for employees interested in applying GenAI. An AI playbook, for example, outlines the concept-to-completion process for executing GenAI ideas and identifies the tools available for doing so, Muthukrishnan said.

General AI training includes periodic AI Days, which include presentations from internal and external subject matter experts, panel discussions and technology demonstrations. Last year, AI Days trained more than 4,600 employes from multiple business units on responsible GenAI use. The bank will hold AI Days quarterly this year, and the first event in 2024 had more than 2,000 attendees, Muthukrishnan said.

The training initiative aims to engage employees across functions and bubble up use cases.

"We believe this is not a technology-only transformation," Muthukrishnan said. "This is a company-wide transformation."

John Moore is a writer for TechTarget Editorial covering the CIO role, economic trends and the IT services industry.