Do Online Symptom Checkers Work, Benefit Patient Triage?

What Are Online Symptom Checkers, How Do They Work?

The rise of “Dr. Google” has led many patients to seek out health information online when they feel symptoms. Online medical research has become more common, with 60 percent of doctors saying in a 2018 Merck Manuals survey that they have noticed more patients coming in with information about their symptoms that they got online.

But the doctors added that the information patients bring in isn’t always accurate, and it can damage the patient-provider relationship as clinicians work to debunk unverified diagnoses and information.

Enter online symptom checkers.

Online symptom checkers emerged to be a more verified way for patients to research their symptoms. Many healthcare organizations, like the Mayo Clinic, have developed their own symptom checkers, while other vendors have worked on tools designed to provide more accurate diagnoses.

These tools ask patients a little bit about themselves—age and gender, for example—and their symptoms. Using different types of AI, the tools spit out a list of possible diagnoses, usually ranked by likelihood. They may also direct patients to their next steps: if they should visit a clinician, in what setting, and how soon.

What Do Online Symptom Checkers Promise?

Online symptom checkers, often embedded in chatbots, are promising because they can efficiently triage patients. When patients describe symptoms of the common cold, the symptom checker should respond by saying patients may ride out the symptoms at home with some over-the-counter remedies. A symptom checker could also alert patients when they should seek urgent or emergency care.

That level of efficiency accomplishes a few key goals. For one thing, symptom checkers have the potential to direct patients to the right kind of care. Data has shown that patients don’t always know where they should access care based on their symptoms. Does a fever warrant a visit to the emergency department? What about chest pain?

In 2017, CityMD published survey data indicating that patients don’t always know when they should go to the emergency department versus the urgent care center. But online symptom checkers should fix this, both by giving patients a possible diagnosis and, in many cases, telling patients how to proceed.

That’s not just good for patient experience and navigation; it’s good for triage, which benefits provider organizations. Symptom checkers should let a patient know to visit their family physician, not the emergency room, when they have certain illnesses, keeping ED crowding to a minimum and avoiding high medical costs.

Moreover, these symptom checkers are automated. In the past, patients might call their family practice to list their symptoms and seek advice about how to proceed. That might still be the best option for extremely complex cases, but a symptom checker that leverages AI should be able to triage a patient exhibiting the common cold. That means healthcare organizations, facing labor shortages, can dedicate their resources to high-acuity issues.

All of these benefits assume that symptom checkers produce the correct diagnosis every time. But data has shown that these tools are still imperfect and that there are limits to their potential.

Are Online Symptom Checkers Accurate?

Although promising for efficiently diagnosing and triaging patients, online symptom checkers are not always accurate.

In a seminal BMJ study, a group of Harvard researchers tested 23 online symptom checkers using 45 vignettes describing patient ailments. Those vignettes fell into three triage categories: emergency care required, non-emergency care reasonable, and self-care reasonable.

The symptom checkers produced lists of possible diagnoses, but they weren’t always accurate, the researchers found. The tools only produced the correct diagnosis at the top of the list 23 percent of the time, and only provided the right diagnosis within the first 20 options 58 percent of the time.

Moreover, the symptom checkers’ ability to triage was variable. Although the tools were somewhat effective for directing patients to emergency care (80 percent success rate), they only identified “non-emergency care reasonable” cases 55 percent of the time and “self-care reasonable” cases 33 percent of the time.

Those figures may prompt some doubt in online symptom checkers, and patients are catching on. Researchers have found that, although patients are open to being screened by symptom checkers embedded in chatbots, they are only satisfied with the care when the bots seem competent.

The data, which zeroed in on COVID-19 symptom checkers and screenings, showed that patients want to know that whoever or whatever is screening their symptoms knows what they’re doing, can follow through on patient needs, and has the patient’s best interest at heart.

It’s much easier for a human to demonstrate those qualities when checking symptoms than a chatbot, the researchers said. But if chatbots can demonstrate that they are effective, they are permissible to patients.

And as AI becomes more sophisticated, it may become easier for chatbots to demonstrate that efficacy, becoming a more attractive option for patients seeking medical information.

What Do Advances in AI Mean for Symptom Checkers?

AI has improved since symptom checkers first came on the market. The likes of ChatGPT might be changing the way the public can interact with symptom checkers, that same group of Harvard researchers wrote in a 2023 Stat First Opinion.

ChatGPT was built upon GPT-3, an AI tool that was unique for its large size (GPT-3 was built on hundreds of gigabytes of online textual data, the Harvard researchers said).

“ChatGPT is a user-friendly version of GPT-3 that includes an easy-to-use chatbox to which individuals can direct questions,” they wrote. “The resulting output is often remarkable; users quickly found they could use ChatGPT for a wide range of applications such as fixing errors in their computer code to writing original, analytical essays.”

The team put ChatGPT to the test and presented it with the same 45 vignettes from their BMJ study. While 23 available online symptom checkers produced a 51 percent accuracy rate, ChatGPT performed better. The AI tool provided the right diagnosis within its first three options 87 percent of the time and properly triaged patients 67 percent of the time.

That compares to the accuracy of actual physicians, who got an 84 percent accuracy rate.

The researchers acknowledged that their vignettes, traditionally used to test medical students and residents, likely aren’t how the typical patient would describe symptoms. Plus, they said they had a small sample size.

Even still, they noted that ChatGPT could help fill a symptom checker void and act as an adjunct for providers.

Healthcare professionals looking at the potential for AI advances to augment symptom checkers should be wary of how they incorporate data about patient health history. It would also be key to examine how AI impacts the patient-provider relationship and how learned bias can impact AI performance.