Are the Updated CMS Hospital Star Ratings Reliable Measures?

The new CMS Hospital Star Ratings methodology, introduced in 2020, is leaving smaller, rural, and critical access hospitals in a lurch, as the new methodology makes for more unpredictable star ratings, according to a new analysis from KNG Health commissioned by the American Hospital Association.

That volatility, which CMS said it did not anticipate when it updated the Hospital Star Ratings methodology, is likely due to the limited number of quality measures that smaller hospitals submit to the agency.

Despite the unpredictability experienced by smaller hospitals, the KNG/AHA report showed stability for larger hospitals that were able to report more quality measures.

CMS uses the Hospital Star Ratings on its Hospital Compare website, allowing healthcare consumers to assess hospital quality and make decisions about where they will receive planned care, like certain surgeries or a birth. For years, industry groups like AHA have admonished the star ratings for making apples to oranges comparisons and for reducing hospital quality to a simple five-star rating.

In 2020, CMS introduced updated star ratings methodologies intended to accomplish two goals. Foremost, it created five measure sets to which hospitals could report quality data: mortality, safety of care, readmission, patient experience, and timely and effective care.

On average, hospitals usually report around 37 measures across each of those five groups, CMS said, but not every hospital needs to report every measure. To receive a star rating, hospitals must report in at least three of the five measure groups, two of which must be mortality and safety of care.

CMS then averages a score within each of those measure groups and calculates a weighted average across those groups. This yields the single hospital summary score.

Each measure group is weighted at 22 percent of the overall score, except timely and effective care, which CMS weights at 12 percent. The agency reallocates the scale if a hospital cannot report in a certain measure group.

The 2020 updates also included the introduction of peer groups, ameliorating the apples-to-oranges problem AHA and its peers have previously identified. This means smaller hospitals with fewer resources will not be scored in comparison to larger hospitals with more resources.

CMS said it did not anticipate the methodology changes to create much volatility in its star ratings reports, suggesting that shifts might happen year over year but that those shifts would be minor.

That was mostly true for larger hospitals, the KNG/AHA report showed. Between 2020 and 2021, 60 percent of all hospitals included in the analysis received the same star rating from CMS. The most common rating was three stars, with 34.2 percent of hospitals receiving such in 2020 and 30.2 percent receiving such in 2021.

Only 36 percent of all hospitals saw a change in their star ratings from 2020 to 2021, and that change was usually only by one star.

But when zooming in on smaller, rural, and critical access hospitals, the KNG/AHA report said there was much more volatility. These hospitals were more likely to report fewer measures to the star ratings.

Comparing application of the old methodology in 2020 to the new methodology in 2021, AHA and KNG said about one in four hospitals with less than 25 percent of reported measures could expect to see a change in star ratings by two or more stars, a considerable variability.

That is not to mention the variability introduced by the peer groups, AHA and KNG continued. The peer groups mean there are now 15 different possible combinations of measure groups. About 74 percent of all hospitals will report across all measure sets, AHA and KNG noted, but for the 26 percent of the hospitals reporting across fewer than five measure sets, there’s room for great variation.

Moreover, there could be differences in which particular quality measures hospitals are able to report. Within a single peer group, two hospitals could report completely different quality measures.

“This means that a hospital’s star rating may reflect a very different set of measure groups,” AHA and KNG explained. “This can be misleading and make it difficult to assess meaningful differences in quality between hospitals with different ratings.”

And in turn, there could be different weighting going on for different measure groups.

“For instance, when a hospital does not report a particular measure group, CMS rebalances the remaining measures reported by redistributing the weight of the excluded measure that is not reported,” AHA and KNG wrote.

A hospital that cannot report on both the safety of care and patient experience (each worth 22 percent of the score) would see the combined 44 percent of that score reallocated to another reported measure. That would inflate the importance of the remaining measures, the AHA/KNG report stated.

AHA said these findings shed skepticism over the entire concept of quality ratings. When CMS first revealed the methodology updates, AHA issued words of support.

“Today’s star ratings update is an improvement that will likely make the ratings more useful for both patients and hospitals,” Ashley Thompson, AHA senior vice president of Public Policy Analysis and Development, said in a statement at the time.

“These changes have made the star ratings easier to interpret, more insightful for hospitals working to improve their quality of care, and more balanced in favor of high-priority topics, like infections,” Thompson said.

But given the KNG findings, AHA said it questions whether reliable, equitable, and useful quality ratings are truly attainable.

“In these most recent changes to star ratings, we see that in attempting to solve one set of challenges around transparency and equitability, CMS has introduced additional challenges,” AHA wrote in the report. “The ratings are still not ‘apples-to-apples,’ as evidenced by the fact that there are so many combinations of groups and measures on which hospitals are assessed. As a result, the ratings are likely to remain unstable for hospitals reporting fewer measures.”

To improve the ratings, AHA advised CMS to consider rating hospitals on individual subject or measure areas; creating peer groups based on other factors than number of measures reported; explaining star ratings to healthcare consumers; and replacing “unstable” measures of rare events with more robust measures of more common patient safety events.

“Patients, consumers and other stakeholders should continue to assess ratings with considerable care, and engage their trusted health care providers in identifying specific measures that matter most to their care needs,” AHA added in conclusion.