Comparison Group Identification Process for the Centers for Medicare & Medicaid Services Financial Alignment Initiative

Identifying an appropriate comparison group is a major challenge in any complex health care evaluations. We describe the comparison group identification process used in the evaluation of the Centers for Medicare & Medicaid Services (CMS) Financial Alignment Initiative (FAI). Comparison group beneficiaries, as the counterfactual for the FAI demonstrations, were used in analyses to measure changes in health care expenditures, utilization patterns, and quality of care. This task was challenging because there are significant differences in health care systems across FAI demonstration states, including differences in state Medicaid programs.

The comparison group identification process has two stages: 1) identifying comparison areas that are similar to the demonstration areas, and 2) identifying the comparison group beneficiaries within those areas who are similar to the individuals in the demonstration areas. In the first stage, we identify many geographic comparison areas at the metropolitan statistical area (MSA) level that are similar to the areas participating in a State demonstration based on area-level measures of health care market characteristics, Medicare and Medicaid spending among Medicare Medicaid eligibles, and Medicaid policy.

In the second stage, all full-benefit dual eligible beneficiaries who are part of the intent-to-treat population under the State demonstration and residing in the demonstration and comparison areas are used to estimate a propensity score model based on beneficiary and area characteristics. After estimating the propensity scores, beneficiaries from the comparison area are weighted using inverse probability of treatment weighting (IPTW) to more closely match the distribution of characteristics of the beneficiaries from the demonstration areas.

Using Minnesota (MN) as an example, the demonstration area consists of all 87 counties in the state, representing 8 MSAs and 1 rest-of-state (non-MSA) area. The combined comparison areas consist of 31 MSAs and rest-of-state areas comprised of 141 counties from 7 states with timely submission of Medicaid data to CMS. The MN demonstration was restricted to those aged 65 years or older. Beneficiaries qualified for the demonstration group if they participated for at least one month during the demonstration period. During the two-year baseline period, all beneficiaries meeting the age restriction and MSA residency requirements were selected for the demonstration and comparison groups.

Pennsylvania and Wisconsin are the two largest state contributors to the comparison group, with 45.3 and 28.9 percent of comparison beneficiaries, respectively. Propensity score weighting using IPTW reduced the disparities of the beneficiary and area characteristics between the two groups to standardized differences of less than 0.12 over each of the two baseline periods and a single demonstration period.

Using the MN demonstration, we illustrate the approach to selecting comparison groups and conducting analyses of group equivalence for the FAI evaluation. The same type of analysis will be conducted for each State demonstration to assess group comparability and the impact of propensity score weighting. Results suggest that our approach can feasibly be applied to evaluate the impact of the FAI demonstrations, and that our two-stage comparison group identification process can be used to construct valid comparison groups for other similarly complex health care evaluations.