Investigating how GenAI can support clinical decision-making

Hurdles to accessing high-quality information at point-of-care

"There's an old credo in medicine that says, 'Half of what they teach you in medical school is wrong. They just don't know which half,'" noted Brad Thompson, M.D., a family medicine physician at Cone Health LeBauer HealthCare, emphasizing that the explosion of medical knowledge -- and how quickly that knowledge could become outdated -- makes effectively utilizing newer insights at the point-of-care challenging.

These difficulties are compounded by the complexities of managing emergent conditions, like COVID-19 infection, or chronic diseases, like diabetes. Primary care providers serve as a critical touchpoint for patients navigating their care journeys, often tasked with sifting through a patient's medical history for clues to home in on a diagnosis or form the basis for referral to a specialist.

Thompson indicated that staying on top of treatment guidelines and emerging medical research is at the crux of this work, but efficiently accessing that high-quality information presents a major hurdle for already overburdened clinicians.

Tools like ClinicalKey AI are designed to streamline the knowledge retrieval process, summarizing evidence-based content sources -- such as medical journals -- in a conversational manner based on a clinician's search query.

Thompson stated that in primary care, a significant portion of a clinician's time with patients is spent figuring out if the symptoms presented align with a common condition or something more serious. In the case of a common condition, providers can often make recommendations or begin setting up care plans at that same visit, and for a more serious condition, the clinician can refer the patient to a specialist.

But Thompson highlighted that there are "gray areas" -- situations in which a patient presents with unique symptoms that might not fall neatly in line with the types of conditions primary care clinicians are accustomed to seeing in their daily practice. In these cases, referring to the most up-to-date medical knowledge is critical to helping that clinician decide how to proceed with a patient's care.

"A lot of things come my way [in primary care], and I'm trying to sort them based on: Is it a common condition that I can help with? Is it something critical I need to deal with right now, or can I -- in that gray space -- get it to a specialist and let it float up there? I've trained hard, and I try to provide good care, but I don't know everything," he explained.

"There's a lot of things in that gray zone, and [the AI] has helped me adjust some of the testing or care navigation to try to tailor it to the patient. It's helping me critique or augment some of my diagnoses," he continued.

But developing an AI model capable of providing clinical decision support in this way is a significant undertaking.

Building clinical knowledge retrieval AI

"Clinicians think that what we have done is trained a large language model with our data, and that's not what we've done … They think GenAI equals LLM, and that's not necessarily the case," stated Rhett Alden, CTO of Elsevier Health Markets, explaining that ClinicalKey AI instead uses retrieval augmented generation (RAG) -- a framework for retrieving data from external knowledge sources.

GenAI tools like ChatGPT are useful for generating, summarizing and understanding new content, but they are often inconsistent when given a knowledge-intensive task or one that requires up-to-date information. This can lead to AI hallucinations, which RAG architectures can help prevent by allowing a model to access information outside of its initial training data.

Utilizing a RAG framework is key to the success of a tool for knowledge retrieval in clinical settings. Unlike traditional chatbots, which are trained on a data set that the model then pulls from when generating responses, ClinicalKey AI uses RAG to pull from a wealth of existing datasets and surface snippets from evidence-based content relevant to a user's query.

The content the tool can access is sourced from Elsevier's own books and journals, alongside other validated clinic journals and sources, like those published by the American College of Cardiology, PubMed and the FDA. Alden emphasized that these journals are often where the guidelines that inform clinical care are published, so the AI provides full access to those materials.

All materials that ClinicalKey AI pulls from are copyright-cleared and reviewed regularly by a team of hundreds of clinicians to ensure that the content is timely, accurate and high-quality. When a user queries the model, it uses natural language processing to understand the query intent and generate a response that cites relevant content from a vectorized database.

"For example, you might ask the question: 'I have a patient that has a kidney infection, but they're also a type two diabetic that's on active short-term insulin management, and they have hypertension, as well as resistance to penicillin,'" Alden noted. "Our system will disambiguate that conversational input and then query against primary source data to retrieve specific paragraphs or subsections of documents that are then summarized by a language-fluent LLM."

Users can then further investigate the full cited sources or see which paragraph from a source was pulled for the AI's response, if they wish.

Alden underscored that this is crucial for driving transparency and trust, which can be a difficult hurdle for clinical decision support tools to overcome.

Elsevier has developed an evaluation framework for the model, in which independent clinicians review roughly 3,000 clinical questions pulled from in-house and open source databases. As part of this process, reviewers regularly monitor factors like completeness of the answer, helpfulness of the response errors -- if there are any errors -- and other metrics.

The tool also has a real-world monitoring process, through which clinician users can report potential inconsistencies or inaccuracies in the model's responses.

This focus on human feedback is intentional, Alden indicated. He explained that many developers rely on positive feedback from users to automate improvement efforts.

"But what we're more interested in is the negative responses," he said. "There, you actually have to have human intervention to understand what was wrong with [the AI's response]. So, we're focused on that because quality and precision are the differentiators that the clinicians care about, and that translates to patient safety in the end."

Prioritizing health equity

Alongside patient safety improvements, Thompson emphasized that the tool can help providers better tailor clinical recommendations and guidelines to their patient's specific case, whether by shedding light on additional lab tests that a clinician might not be familiar with or by providing further insights into novel treatment approaches.

Personalized medicine is becoming increasingly popular as data analytics capabilities, genomics and other innovations in the healthcare industry have become more widespread. But the potential for tools to help individualize care must be balanced alongside guardrails to ensure that they do not perpetuate health disparities.

Concerns around bias are one of the major drawbacks of healthcare AI, as human prejudices can be inadvertently baked into a model during development. Biases can also crop up in medical research, whether through the underrepresentation of certain groups in clinical trials or the use of race as a biological construct in algorithms tested in clinical studies.

In the case of ClinicalKey AI, Alden indicated that prioritizing health equity looks like considering the three sources of potential biases: the data, the way the tool summarizes that data in its responses and the user.

Managing data biases comes down to having a mechanism for the editorial review of studies prior to publication, alongside ways to recall articles or refute the findings later on, if necessary. Peer-reviewed journals have such mechanisms in place, reducing the likelihood that biased materials will be published.

The process of combing through the tool's database for quality control helps ensure that its responses are unbiased, but biases within users are nearly impossible to contend with.

You shouldn't be disadvantaged because you happen to live in a low-income region and don't have access to the kind of clinicians that Mayo Clinic or Cleveland Clinic have. You should be able to use tools that upskill those clinicians with appropriate information to address your needs. So, that's the vision of this kind of tool: to really offer access -- the broadest access possible -- to quality information. That's good for the patient, and it's good for the clinician.

However, Alden noted that the AI can help tackle these by promoting access to high-quality clinical knowledge.

"This kind of technology gives the clinician much easier access to correct information and offsets the risk of giving them bad information," he explained, stating that the tool is designed to consider "patient context" in its responses, as long as that information is provided by the clinician.

He illustrated his point through vignettes of patients taking a neurological drug with potential contraindications with hormonal birth control. If one asked the tool what precautions should be given to a patient taking said neurological drug, it would provide a more generic overview, whereas if the query included that the patient is female, the response would be more likely to highlight the birth control contraindication.

Alden underscored that including a detail like gender identity in the query is important when providing care for marginalized groups, noting that if you posed this query to the tool with the added context that the patient is transgender and assigned female at birth, it could provide an even more granular response.

Being able to niche down within a query to pull more individualized insights is valuable when attempting to provide high-quality care for members of marginalized groups, but this can present a challenge.

"I would argue that the clinician would be unable to ask those questions very easily without our tool. So, part of this is an awareness on the clinician's part of dealing with marginalized groups and understanding how to leverage the tool properly," he stated.

"If it's a fully qualified question, we can give very detailed responses that address marginalization and systemic bias in a way that's appropriate, as a lot of clinicians are not trained at all on transgender patient care or other issues," he continued.

Since the onus of asking a qualified, appropriate question of the tool lies on the clinician, Elsevier is exploring the possibility of providing training and guidance around how to use the tool properly and mitigate potential biases.

While these efforts to reduce bias are not necessarily foolproof, Alden stated that they are likely to encourage meaningful progress in improving health equity by helping providers be more cognizant of potential bias concerns.

Alongside real-world monitoring and clinician feedback, he noted that hearing from patients is also crucial for advancing equity efforts.

"Clinicians aren't the final arbiters of bias," Alden stated. "We actually need to hear from LGBTQ+ groups, Black communities and other potentially marginalized groups in terms of how they see this tool working."

By bringing together a variety of stakeholders, clinical knowledge retrieval tools have significant potential to democratize care quality, he continued.

"You shouldn't be disadvantaged because you happen to live in a low-income region and don't have access to the kind of clinicians that Mayo Clinic or Cleveland Clinic have," Alden said. "You should be able to use tools that upskill those clinicians with appropriate information to address your needs. So, that's the vision of this kind of tool: to really offer access -- the broadest access possible -- to quality information. That's good for the patient, and it's good for the clinician."

Shania Kennedy has been covering news related to health IT and analytics since 2022.