ChatGPT Capable of Answering CBME Microbiology Questions

Researchers writing this month in Cureus have found that ChatGPT is capable of answering first-order and second-order questions in line with the competency-based medical education (CBME) curriculum for microbiology with significant accuracy.

ChatGPT has been trained to respond to queries on a wide variety of topics, including US Medical Licensing Exam (USMLE)-style questions, a task in which the chatbot achieved a 60 percent score. This score, considered a passing grade on the actual USMLE, has led some researchers to conclude that ChatGPT may have potential for use in medical education.

However, such applications for the technology have not been fully explored in microbiology, the Cureus article’s authors argued. In particular, ChatGPT’s ability to accurately answer first- and second-order knowledge questions had not yet been explored, leading the researchers to investigate its potential.

To do this, the authors prepared six first-order and six second-order knowledge questions according to the eight modules in the National Medical Commission-recommended CBME curriculum for microbiology, resulting in 96 questions.

The researchers defined first-order questions as those that are more straightforward, asking for factual information or seeking a direct answer, while second-order questions were characterized as more complex, requiring higher-level thinking and interpretation of information, opinions, and predictions based on evidence.

These questions were then reviewed for content validity by three expert microbiologists and fed to ChatGPT by a single user. The chatbot’s responses were recorded and scored on a scale of 0-5. The average of the scores for each question was taken as the final score in the analysis of ChatGPT’s performance.

The median score for all 96 questions and the first-order questions was 4.17, while the median score for second-order questions was 4. However, the researchers noted variation across the eight topic categories, with inconsistent median scores across different topics.

Overall, ChatGPT achieved an accuracy of about 80 percent. The authors indicated no difference between the model's capability of answering first-order and second-order knowledge questions.

They concluded that these results show that ChatGPT is capable of accurately answering both first- and second-order questions related to microbiology. They noted that the technology might have potential as an effective tool for automated question-answering, which may assist medical students in self-directed learning. However, improvements to ChatGPT and other large language models are needed before they are suitable for academic use.

Concerns about ChatGPT’s use are also prescient for clinicians and researchers, particularly those in the oncology field, as online health information-seeking becomes a growing concern.

Researchers shared last week that while ChatGPT provides accurate information when asked about common cancer myths and misconceptions, these answers could be interpreted incorrectly and, as a result, negatively impact patient decision-making.

The research team indicated that misinformation and harmful information about cancer are prevalent online, presenting a major challenge for clinicians as more patients turn to the internet, and chatbots, to find information about the disease.

After comparing responses from ChatGPT and the National Cancer Institute (NCI) to questions pulled from the NCI “Common Cancer Myths and Misconceptions” web page, the researchers found that despite the chatbot’s accuracy of 96.9 percent, reviewers felt that ChatGPT’s language could be indirect, vague, or unclear, which could lead to patient harm if not addressed.