Paid medical chatbots better at health literacy than free ones

Not all medical chatbots are created equal. Some, according to a JAMA Network Open study, account for patient health literacy better than others.

The report, which compared the paid and free versions of ChatGPT, showed that the version behind the paywall provides more readable health information than the free one. However, after adding, "Explain the following at a sixth-grade reading level," researchers detected little difference in readability.

These findings come as healthcare experts work to better understand how AI-based chatbots can augment the patient experience. This latest study shows that the paid and free versions of chatbots could potentially exacerbate health disparities in terms of health information access and health literacy.

Chatbots like ChatGPT have become key parts of the healthcare conversation. These tools have shown promise in improving patient access to health information, with preliminary studies indicating that the answers provided by chatbots heavily consider patient health literacy levels.

But even still, chatbot quality can vary. ChatGPT, one of the most commonly used AI-based chatbots, has both free and paid versions of the technology, suggesting different tiers of efficacy.

This latest study tested that, comparing the free and paywalled versions of ChatGPT for both readability using the Flesch Reading Ease score and consumer health information quality using the DISCERN instrument.

The researchers fed both versions of ChatGPT the five most popular queries related to breast, lung, prostate, colorectal and skin cancer between 2021 and 2023 and found discernable differences between the two in terms of readability.

While there was no difference in consumer health information quality, the researchers said the paid version of ChatGPT provided answers with higher readability scores than the free version (52.6 versus 62.48 on the 100-point scale). Notably, the readability scores of either version of the chatbot were suboptimal, the team said.

However, the researchers were quickly able to level out those scores by prompting the free chatbot to explain concepts at a sixth-grade reading level. This increased readability scores for the free chatbot to 71.55, meaning the free version of ChatGPT performs better than the paid version, so long as the user prompts it to give responses at a sixth-grade reading level.

When both the free and paid versions of ChatGPT were prompted to provide answers at a sixth-grade reading level, the paid version scored a 75.64 readability score, still outperforming the free version.

Despite the positive effect of asking ChatGPT to simplify its answers, the study showed an overall problem with AI-based chatbots and health literacy. The readability scores before prompting for a sixth-grade reading level were suboptimal, the researchers said. The median reading level of these responses was around 12th grade.

Although the second round of queries asked for a sixth-grade reading level, the researchers said those responses still came in more like an eighth- or 10th-grade reading level.

That might have been due to chatbot confusion, the researchers posited. The chatbot might have interpreted the request as the user being in sixth grade, not asking for a sixth-grade reading level of the Flesch scale. For example, after prompting for responses at a sixth-grade reading level, ChatGPT recommended the user discuss cancer topics with an adult, such as a parent or a trusted teacher.

In addition to the middling readability, these findings call into question health equity. If the paid version of ChatGPT provides more easy-to-understand health information, an individual with the disposable income to purchase the paid chatbot might have an advantage in achieving health and well-being.

"These findings suggest that chatbots have the potential to contribute to cancer health inequities when responses are presented above the recommended sixth-grade reading level for consumer health information, particularly when the general public is seeking cancer information with the unpaid version of a chatbot," the researchers said.

As long as chatbots are still providing health information at a higher reading level, it might be incumbent upon clinicians to coach patients in best practices for medical research and chatbot use. Clinicians should consider advising their patients to query information explicitly at a lower reading level, the researchers concluded.

Sara Heath has covered news related to patient engagement and health equity since 2015.