Testing the Robustness of Health Care System Rankings

The rankings of national, state, or county health care systems, which aggregate different attributes of population health into one summary measure, garner much attention from public health officials and the media.  These rankings often suffer from two main methodological issues.  First, the rankings often use a linear aggregation function, computing a weighted mean of the attributes where the weights are determined according to each measure’s “importance” in a health care system.  However, a linear aggregation function is akin to assuming that the different attributes are perfect substitutes.  Second, counting multiple attributes capturing similar variation in the distribution of population health effectively places more weight on these sources of variation.  In this paper, we propose a new methodology for ranking health systems based on generalized multivariate relative entropy first developed in Maasoumi (1986) and widely used in the measurement of welfare. This methodology enables us to vary assumptions about the substitutability or complementarity of attributes and account for the double counting of attributes.  We use data from the 2012 version of America’s Health Rankings of U.S. states and find significant variation in health care rankings depending on the assumptions used.  Our results have broad implications for the measurement of health system performance, and our methodology is applicable to many other issues in health such as health inequality, intergenerational health transmission, and the measurement of hospital and physician performance.