AI Shows Promise in Improving Orthopedic Surgery Documentation

Research presented at the 2023 American Academy of Orthopaedic Surgeons (AAOS) Annual Meeting evaluated multiple clinical documentation modalities to determine the overall quality and how long each takes to capture a patient encounter, revealing that artificial intelligence (AI)-based virtual scribe services show promise, but require additional verification.

The study aimed to examine how the administrative burden on clinicians caused by EHR documentation can be addressed.

"In our practice, we created a task force to better understand and correct physician burnout to study what we know to be the top reason for burnout — patient documentation," said Michael Rivlin, MD, an orthopedic surgeon at the Rothman Institute and associate professor at Thomas Jefferson University, in the press release. "We wanted to look at ways to maximize the physician's workload at the maximum level of their license and remove burdens that can lead to burnout by finding methods to outsource certain tasks, such as documentation, as this can be time consuming and redundant."

The researchers used four modalities to document common orthopedic encounters in hand surgery patient visits: an AI-based virtual scribe service run on a tablet and tasked with extracting everything said in the room, a medical scribe either physically in the office visit or participating virtually to transcribe the patient encounter, a transcription service in which the clinician records an audio file about the patient visit and sends it to a third-party company for transcription, and a voice recognition mobile (VRM) application, an EMR-integrated tool that types the words being said based on voice recognition.

During the study, three orthopedic hand surgeons evaluated 10 standardized 'patients' with prewritten clinical encounters or 'vignettes.' Clinical documentation during the encounters was initially performed using the AI-based scribe and the medical scribe and, afterward, with the VRM and transcription service.

"Our physicians who were not involved in the documentation acted out these vignettes and each scenario contained an element of distraction to determine if the AI would be thrown off by various nuances that might occur during a clinical visit — such as a parent and a minor sharing their thoughts, or a patient interjecting a story about a friend's experience with hand surgery in the middle of providing an update on their own surgery," Rivlin explained.

Using this study design, the researchers documented 118 clinical encounters, including 30 with the AI scribe, 30 with the VRM, 28 with the transcription service, and 30 with the medical scribe. Any clinical notes generated during each encounter were categorized as acceptable or unacceptable and given a letter grade of A, B, C, or F based on an eight-point scoring system. Following this process, an attorney reviewed all notes to assess medical legal risk.

The research team found that all modalities achieved similar levels of high performance but found key differences between them.

A significant limitation of the AI modality is that it scored lower than the others on the question “is the plan correct?” The AI was able to gather most of the implied and verbal elements within its documentation, but its performance was deficient compared to the human medical scribe and required a manual edit of the ‘plan’ section of the documentation.

The AI-based scribe’s need for additional correction and verification is a major drawback to its use in the clinical setting, but it has significant potential, the researchers noted.

"The AI-based virtual scribe service is a promising tool to help decrease documentation burden without significantly lowering the quality of documentation compared to transcription and voice recognition software services," said Rivlin. "While AI has some limitations, it continues to improve as the technology advances. These results create a palette of options for physicians to compare outputs should they want to explore new modalities."