HIMSS25: Navigating the cost-benefit dilemma of health AI

Separating the shiny objects from the real deal

According to Nigam Shah, MBBS, PhD, chief data scientist for Stanford Health Care, the cost-benefit analysis of AI tools goes beyond their primary capabilities. Take AI-powered predictive models, for instance.

"The point is that the model just gives you a prediction or risk stratification, if you will, but the benefit comes from the response to action," he said during a session at HIMSS25 in Las Vegas. "And so, in that setting, we have to study the model, we have to study the capacity to act, we have to study the cost and benefit of the action itself."

With AI-driven predictive analytics, a critical question for health system leaders is: what is the utility gained from a prediction? Based on a health system's capacity for action, the utility could be hours saved, money saved or lives saved. However, if the capacity for action is limited, the utility will be as well, diminishing the tool's return on investment. Simply put, a predictive AI model to flag sepsis risk in patients is pointless if the health system does not have clinical staff who are able to intervene quickly.

In some cases, the benefit gleaned from an AI tool differs from the expected benefit, which means that health system leaders must thoughtfully define and verify benefits when assessing new AI tools.

For example, when Stanford assessed the impact of automated in-basket response technology on productivity, they found that the technology did not save time and reduce after-hours work as expected; however, there were other advantages.

"Our doctors love it, their cognitive burden went down, and they're happier," Shah said. "100% true. [But] that's not what we went in with. If you did an RCT in this way, the FDA would never approve your product because we changed the outcome after we finished the study."

He further emphasized that applying the efficiency/productivity lens to genAI may have counterintuitive results. Improving productivity means that physicians may be able to see more patients, but this could add to physicians' workloads in the long run.

The point is that the model just gives you a prediction or risk stratification, if you will, but the benefit comes from the response to action. And so, in that setting, we have to study the model, we have to study the capacity to act, we have to study the cost and benefit of the action itself.

Thus, health systems must focus on "the achievable benefit" of the tool in question when conducting cost-benefit analyses, Shah said.

Exploring differing approaches to cost-benefit analyses

The achievable benefits of AI tools vary according to the use case. For example, physician well-being is a significant benefit for health systems implementing AI to ease administrative tasks.  

In an interview at HIMSS25, Providence Chief Medical Information Officer J. S. Smitherman, MD, said that implementing AI across a large organization is a significant financial investment. But the health system's cost-benefit analysis takes into account more than just productivity; it considers the ROI gleaned from keeping physicians in the workforce longer.

"Ambient [AI] tools make it easier for them to practice, and they're willing to work for another year or two," said Smitherman. "In aggregate, we have almost a quarter of the doctors in the United States over age 65. And with an aging population, that is a major crisis. We need to keep a lot of those doctors practicing if we want to have [healthcare access]."

And it's not just the large numbers of retiring physicians. Smitherman said that many younger physicians are struggling to fit the demands of a full-time job into the time frame allotted for a full-time job.

With nearly 50% of the physician workforce reporting burnout, ambient and other types of AI that ease that cognitive burden may be worth the cost. The positive impact on physician well-being at Providence appears to be worth the price, with robust physician adoption of AI tools. Smitherman noted that adoption is a vital measure of whether technology implementation is working as intended.

"Doctors are sort of notoriously resistant," he said. "So, it's got to work. They're going to vote with their feet. If it weren't working, the doctors wouldn't use it."

Northwestern Medicine's approach to the cost-benefit analysis of AI borrows from Silicon Valley's "fail fast" adage. Hannah Koczka, vice president of ventures and innovation at Northwestern Memorial Hospital, noted that the health system brings in many technologies and quickly tests them on a very small scale.

"It's more like a proof of concept -- so something that we can maybe do in a matter of weeks or a matter of only doing something in one department or one area before we then see if we can prove it out," Koczka said. "And then we might do something at a much larger scale, like a six-month pilot or something like that."

This agile approach to assessing AI technology allows the health system to move through several technologies before settling on a tool that meets its needs and is worth the high price tag.

Health system leaders are also hopeful that as AI becomes ubiquitous, costs could come down.

Smitherman stated that even now, AI tools that take on low-hanging administrating burdens, like reviewing in-basket messages, are not as costly as they used to be.

"As these [AI] models get better and faster, that's improving," he said. "But still, there's going to be a lot of places where we do have to choose to make investments; but once again, because they're different products, we just kind of have to evaluate the various ROIs. I think some of the ones where [there is a] clinical gain might be the toughest, right? Because nobody wants to put an exact number value on anybody's life, but if an AI tool can potentially make something safer, the system certainly is willing to invest in that."

Anuja Vaidya has covered the healthcare industry since 2012. She currently covers the virtual healthcare landscape, including telehealth, remote patient monitoring and digital therapeutics.