AN APPROACH FOR MAXIMISING EXTERNAL VALIDITY IN COST-EFFECTIVENESS ANALYSES (CEA)

Cost-effectiveness analyses (CEAs) use data from randomized controlled trials (RCTs) to maximize internal validity. However, RCTs tend to include atypical patients and centres, and so CEA results are subject to sample selection bias. To reduce this bias, observational data can be used to adjust the RCT-based estimates. We present an approach that defines the underlying assumptions, which include that the selection of RCT participants is according to observable characteristics. We then develop a new method that makes a less restrictive selection on observables assumptions. We critically examine the alternative methods in case studies of high policy-relevance. These include a CEA of Pulmonary Artery Catherization (PAC) using an RCT (n=1,014, 65 centres), the Control of Hyperglycemia in Paediatric Intensive Care (CHiP) trial: (n=1350, 15 hospitals), and a CEA of Telehealth, the remote exchange of data between patients and healthcare professionals in primary care (n=3000, 179 GP practices). For each study we exploit longitudinal observational data on case-mix and outcomes for individuals in the target population.

We decompose sample selection bias into observable or unobservable differences between the RCT and the target population. The paper derives those assumptions required to identify average treatment effects for the target population from the RCT data. We provide placebo tests which follow formally from the identifying assumptions, and can assess whether or not the assumptions hold, for the target population, and subpopulations of interest.  These tests use observational data to establish whether unbiased estimates in the target population can be inferred from a given RCT..  Such placebo tests assess the non-equivalence of reweighted outcomes following either ‘treatment’ or ‘control’ in the RCT versus the outcomes for the corresponding treatment groups in the target population. These placebo tests are only passed if the identifying assumptions hold, and there is sufficient power to detect a difference in outcomes across settings. We then consider an alternative design, similar to difference-in-differences (DiD), which adjusts for some unobserved as well as observed confounding between the RCT and the target population. This design can provide an estimate of the sample selection bias, after adjusting for the selection into the RCT according to observed characteristics.

For each case study we adjust the RCTs estimates, to the observed characteristics of the target population using alternative reweighting and regression techniques. We report cost-effectiveness overall, and for subgroups defined a priori.While the placebo tests are passed for the PAC and CHiP case studies, for the Telehealth case study the placebo test results  show that the adjusted outcomes for the control patients in the RCT differed to those for patients receiving usual care in the target population. We then apply the DiD approach to re-estimate the effectiveness and cost-effectiveness of Telehealth in the target population.

We conclude that the approaches developed can help maximise the external validity of CEA. The placebo tests presented can identify when the requisite assumptions are not met. The DiD approach is useful in adjusting estimates of program’s effectiveness and cost-effectiveness in the target populations to reduce sample selection bias.