Vendor-Derived EHR Use Measures Present Research Limitations

As the number of studies using event logs to observe EHR use continues to grow, inconsistent measure definitions from vendors and investigators pose challenges for clinical research, according to a study published in JAMIA.

Researchers reviewed studies that employed measures of EHR use derived from EHR event logs.

They found that researchers use vendor- and investigator-derived measures independently—just one study out of 151 employed both. Additionally, the study authors found that researchers use vendor- and investigator-derived criteria for different kinds of research.

The authors noted that while vendor-provided measures remove many barriers to conducting log-based research, they are limited in scope.

Studies using vendor-derived measures focused almost exclusively on ambulatory physicians and APPs. On the other hand, studies employing investigator-derived measures examined EHR use in ambulatory and acute care by a broader range of users, including physicians, APPs, nurses, technicians, students, and scribes.

“While some vendors provide measures of EHR use for these roles, the lack of studies reporting them suggests a lack of measure awareness or accessibility,” the authors wrote. “Until vendor-provided measures of EHR use are more widely available and accessible for all EHR users, investigators will need to continue deriving custom measures for some users.”

In addition, studies employing vendor-derived measures were more likely to raise concerns about measure opacity and measure availability for certain clinical roles.

“The research community should continue to develop and adopt standardized measures of EHR use, such as the seven measures of ambulatory EHR use proposed by a national research network of EHR log researchers, and to work with vendors to shape vendor-derived measures as they become de facto standards,” the study authors said.

The researchers noted that vendors and investigators have unique roles to play in measure development.  

“EHR vendors are well positioned to curate generalizable measures of the duration and volume of EHR activity agnostic to specific workflows,” the authors wrote. “Some vendors have provided the methodological decisions behind their measures to customers or referenced them in studies, but these methods are inconsistently reported in the studies that depend on them.”

For instance, only 59 percent of the studies that reported active EHR use duration described how active use was defined.

“Publishing vendor’s validation studies, which have been referenced in several studies but not explicitly reported, would also help ensure accurate accounting of log-based measures,” the authors added. “Investigators in turn are uniquely positioned to validate the measures they derive from event logs, particularly those of workflow and team dynamics which may be workflow or site-specific.”

Ideally, validation efforts by vendors and investigators will include explorations of whether measures are equally valid across different kinds of clinicians.

The review has several limitations that future work could address.

“First, it considered measures derived from EHR event logs but excluded studies based on related data such as logs from other health information technology (e.g., telephone logs), and timestamps stored in patient records (e.g., check-in time),” the review authors wrote. “Research analyzing these data may have distinct aims, measures, and methods compared to the literature surveyed in this review.”

Additionally, article abstraction is subjective, and a single author largely performed it in this review.

“We mitigated potential bias by using a coding scheme derived from a prior review, and by iteratively revising and validating the coding scheme through independent coding of the same articles by two authors,” the researchers pointed out.