With artificial intelligence and machine learning, KPIs are crucial

Building an AI-conversant enterprise

DBS Bank is in the midst of an enormous digital transformation effort, one that includes an aggressive plan to implement artificial intelligence and machine learning. Part of the near-term effort is to build an enterprise AI competency.

"We want to train the top 250 executives in the firm to become AI conversant," said David Gledhill, group CIO and head of group technology and operations at DBS Bank. "We want to train 200 people to be what we call AI translators."

By conversant, Gledhill means training executives to understand machine learning techniques at a high level, as well as knowing enough to "intelligently question what a model is and what it can do," he said.

David Gledhill

The numbers he shared are more than a want, they are key performance indicators for the year, and they are tangible enough to give employees a realistic goal without drowning them in the vision thing. "We don't want to go too far, too soon because we think that's just going to completely alienate people," Gledhill said.

And, ultimately, this executive cadre is an important part of a larger strategy to grow the company and to foster a reputation as a premier digital bank. Gledhill described the education efforts as "phase one" of a multiphase plan. Phase two metrics are being hashed out now but will focus on "how to solve process at scale." That could include KPIs that measure process automation as well as targets for company growth.

Phase three is still a work in progress but will likely focus on observation and prediction. "A goal that one of our teams there has already set is how do I detect a million problems before they happen," Gledhill said.

The customer comes first

Gledhill said he sees artificial intelligence and machine learning as "just another tool in the toolbox" and the implementation of the technologies as an extension of the data culture already in place.

What's missing is employee training and a supporting workflow design. When Gledhill and his team began asking what it means to design for AI, they helped guide the conversation to focus on "data capture, data governance, instrumentation, and also the way you design a process and what is applicable to AI," he said. "Training people to understand what that actually means is kind of the challenge and the opportunity."

At Kayak, artificial intelligence and machine learning are also seen as tools in the toolbox, but KPIs that measure their usage simply don't exist. The company, which aggregates prices on flights, hotels and car rentals, has been using artificial intelligence and machine learning behind the scenes for years.

Giorgos Zacharia

"At the end of the day, it's about measuring user experience," said Giorgos Zacharia, Kayak CTO and chief scientist who holds four math and computer science degrees from MIT. "If machine learning is the right tool, you use it. If it provides improvement, you use it."

When pressed to share UX KPIs that might influence developers to choose artificial intelligence and machine learning tools over traditional coding techniques, Zacharia insisted the company just doesn't think that way. "We basically have one KPI: Have we helped users find the flight, hotel or car they were looking for?" he said. "Whatever tools you use underneath, they have to drive toward an improvement for that."

Indeed, customer experience appears to influence just about every decision at Kayak. Even when talking about security measures, latency and fraud detection, Zacharia brought his point back to user experience. "Our own suppliers of information -- airlines, online travel agencies, et cetera -- love our data," he said. "Some of them actually hire bot scrapers to find out their competitors' prices. So protecting the website from malicious -- and non-malicious -- bots is very important because it affects user experience."

A shared goal

The distinction between the detailed, multiyear plan from Gledhill and the laser-like focus on customer experience from Zacharia drove Michael Schrage, MIT researcher and panel moderator, to pit the two against each other. "What advice would you offer [Gledhill]?" he asked Zacharia. "Do you think he's being too process-oriented on this?"

"Maybe. But it's a bank," said Zacharia, drawing a hearty laugh from the audience.

Gledhill didn't let the comment slide, asserting that the financial institution's goal to solve a million problems before they happen is directly tied to customer experience. He described DBS Bank as being "on this journey" to make customer experience less of a passive endeavor and more of a predictive one.

"As we evolve our product set," he said, "we've kind of gone on this journey of customer experience from customer feedback, which simply doesn't work out well, to design thinking to what we're now exploring, which is this whole area of customer science," he said. He said it was important to understand the science of customer behavior, to find areas where customers struggle and to determine what great service looks like.

We are at war, and, as my boss keeps reminding me, digital and technology is the way we're going to win or lose.

Rather than designing for AI, DBS Bank's developers will eventually design for what Gledhill called "customer ops." That includes finding the friction points customers might experience, figuring out how long a customer is likely to wait for a response before moving on to the next transaction, aligning KPIs to a customer ops dashboard in an effort to stay one step ahead of customers, and using cutting-edge technologies like artificial intelligence and machine learning to make it happen.

"We are at war, and, as my boss keeps reminding me, digital and technology is the way we're going to win or lose," Gledhill said.