So Many Hospitals, So Little Information: How Hospital Value Based Purchasing is a Game of Chance

Since 2005, The Centers for Medicare and Medicaid Services (CMS) have published Hospital Compare, a hospital quality ranking system that calculates hospital-specific performance on specific clinical metrics. A small financial penalty applied to hospitals that did not submit quality information, leading to near-universal hospital participation in the program. CMS publishes these scores so that patients can make informed decisions when choosing a health care provider by directly comparing inpatient facilities.

The published Hospital Compare scores document changes in clinical treatment through time. As a result of financial incentives to report and publicly available performance information, hospitals were able to vastly improve their performance on these measures in a short period of time. The widespread improvement resulted in tightly bunched distributions of published scores around high levels of performance in the years following initial score publications.

With the introduction of the Patient Protection and Affordable Care Act, Hospital Compare took on a new role: hospital reimbursement calculations under the Value-Based Purchasing (VBP) program include a hospital’s performance on these measures.  Under VBP, a CMS withholds a small percent of the total reimbursement pool (currently 1.5 percent of payments and anticipated to be 2 percent of payments), and then redistributes the funds based on how hospitals perform relative to other hospitals and relative to their own previous scores.

The payment redistribution scheme depends heavily on the ability of the quality scores to rank hospitals relative to one another.  As the score distributions bunch together and hospital rankings become statistically indistinguishable from one another, the VBP system begins to resemble a lottery for reimbursement redistribution. 

This study uses measures of statistical indistinguishability in ranking systems to measure how much of the distributions that underlie the VBP calculations are indistinguishable from the relevant cutoffs for VBP point generation.  In essence, we calculate the amount of the distributions used for VBP point calculations that are subject to a reimbursement lottery.

We develop a measure of the level of statistical indistinguishability in ranking that is present in an entire distribution: similar to a Gini coefficient, our measure quantifies the statistical noise in a ranking distribution.  We use this statistic to track the level of noise there is in the rankings from the distributions of quality scores over time.  Finally, we evaluate the appropriateness of using tightly packed distributions to rank hospitals.  We conclude that while quality scores serve the purpose of incentivizing hospitals to adhere to clinical best-practices, the positive incentive also creates score distributions that are largely inappropriate for use in hospital rankings and payment