Do Medicare Advantage Star Ratings Actually Capture Quality?

CMS assesses the quality of the Medicare Advantage (MA) program using a 5-star rating system, but the validity of these ratings remains unknown. Higher rated contracts tend to enroll healthier, and less socio-demographically disadvantaged beneficiaries and enrollees with worse health status disproportionately disenroll. It’s unknown whether contract performance ratings reflect a better quality of care, or if they are driven by favorable selection.

Contracts rated 4+ stars receive up to a 5% payment bonus to their capitated payments for each enrollee. It has been recently reported by the Wall Street Journal that MA companies engage in contract consolidation to maximize these payments. Under current rules, a company with multiple contracts may choose to “consolidate” one contract into another, effectively moving all enrollees from one contract into another already existing one. The company would then be eligible to receive bonus payments for the transferred enrollees if the new contract is rated 4+. As contract ratings are endogenous with enrollee selection, we employ a novel instrument based on contract consolidation to estimate the effects of higher star ratings on outcomes. This happens without enrollee input and presents a shock to an enrollee’s contract that is plausibly exogenous from initial enrollee selection. We find 1,064,933 enrollees who were enrolled in <4 MA plan in 2014 that were consolidated with another MA plan in 2015.

We use the Medicare Beneficiary Summary file to track enrollment over time and to identify contracts that consolidate from one year to the next. We focus on enrollees who started in <4 star rated contract in 2014 in order to make a better comparison between enrollees who were consolidated and those who were not. We calculate patient level quality outcomes from 2014 and 2015 Healthcare Effectiveness and Information Data Set (HEDIS) files. We also calculate post hospitalization readmission, and mortality rates, and assess the quality of hospital networks from MedPAR hospitalization data.

We include observations for each enrollee in 2014 and 2015. In our models we include patient characteristics including age, dual status, basis of eligibility, race/ethnicity, and chronic disease. In the first stage, we estimate the role of consolidation on an enrollee’s 2015-star rating to instrument for the relationship between star ratings and enrollee 2015 outcomes using standard 2SLS models. We test alternative specifications of including beneficiary or 2014 contract fixed effects in the model to better isolate the relationship between star ratings and outcomes.

In our first stage, we find that contract consolidation is strongly correlated with 2015-star rating with an F-statistic of over 10,000 indicating a strong instrument. In both OLS and IV models, we find that, star rating may be associated with improved access to high quality providers and some gains in quality outcomes. Our results indicate that the star rating system as it is currently constructed may be capturing some important characteristics of MA contracts.