234
COMPARISON GROUP SELECTION WITH ROLLING ENTRY IN HEALTH SERVICES RESEARCH
Non-randomized healthcare demonstrations commonly enroll participants on a rolling basis; however, rolling recruitment complicates impact evaluation for researchers. This research presents a comparison group selection methodology for interventions with rolling participant entry over time. A difficulty in evaluating rolling entry interventions is that a suitable “entry” date is not observed for non-participants. Our method, called rolling entry matching, assigns potential comparison non-participants multiple counterfactual entry periods which allows us to match participant and non-participants based on data immediately preceding each participant’s specific entry period, rather than using data from a fixed pre-intervention period.
STUDY DESIGN
Rolling entry matching is demonstrated using a synthetic population of individuals who resemble Medicare fee-for-service patients in terms of age, race, spending, inpatient visits, ED visits, chronic conditions, dual eligibility. We impose an intervention effect for participants in the data and then select a matched comparison group using rolling entry matching. We compare the estimated treatment effect obtained to the “true” treatment effect.
Rolling entry matching requires three steps. First, a quasi-panel dataset is constructed containing multiple observations of non-participants (one for each entry period). Participants enter the data once in the baseline period immediately preceding their unique entry into the intervention. Time-varying covariates (e.g., health conditions, spending, utilization) are dynamic for each entry period’s non-participant observations. Second, a predicted probability of treatment is obtained for participants and non-participants (e.g., through propensity score matching). Finally, the pool of potential comparisons for each participant is restricted to those that have the same “entry period” into the intervention. The matching algorithm then selects the best matched comparison(s) for each participant from the pool of non-participants with the same entry period.
RESULTS
Preliminary analyses indicate that rolling entry matching generates intervention effects that are similar to the true intervention effect. Further work will evaluate how comparison data cloning performs relative to other comparison group selection methodologies.
CONCLUSION
Techniques for selecting comparison groups in the context of rolling entry include using a single pre-intervention period for all participants, block/cohort matching, and optimal risk set matching; however, these methodologies may exclude data immediately preceding entry from the matching process or may be hindered by sample size. Rolling entry matching is a novel approach to comparison group selection that utilizes data from the period immediately preceding entry, which is important for interventions in which a health event is a prerequisite for entry. Furthermore, rolling entry matching is less susceptible to sample size limitations.
In the post-ACA landscape, health care interventions are being implemented on a rolling basis to maximize enrollment and impact. Evaluation methodology must also advance to provide rigorous and valid impact estimates. Reliable estimates hinge on valid comparison groups acting as the counterfactual case for the treatment group in absence of the intervention. Because rolling entry matching selects comparison individuals that are similar to intervention individuals in the period immediately preceding entry, the comparison group has greater exchangeability with the intervention group and inferences drawn about intervention effects are less vulnerable to potential bias than other commonly used methods.