Using Disaster Induced Closures to Evaluate Discrete Choice Models of Hospital Demand

Economists frequently evaluate counterfactuals by estimating structural models with behavioral parameters. This approach typically involves modeling agents' behavior, using econometrics to identifying policy invariant parameters of that model, and using these parameters to make predictions in changing environments. This approach to empirical research is common in health economics, where it is used to evaluate a range of policies.

Estimating behavioral parameters that accurately reflect behavior after a change in environment is critical to obtaining credible counterfactuals. Consider the example of hospital demand models, which we study in this paper. According to the assumptions of the underlying economic model frequently used in hospital demand estimation and for counterfactuals, patients' utility for a given hospital should not change with a change in the choice environment. The invariance of these parameters is what allows us to model counterfactual changes that are used to predict price effects of hospital mergers and the impact of narrow networks - both of which involve counterfactuals that change consumers' choice sets.

In this paper, we use two approaches to evaluate how models of hospital demand perform in a changing environment. First, we consider the invariance of the models' behavioral parameters. Second, we consider the predictive accuracy of these models in making counterfactual policy predictions.
We use the closure of hospitals from four natural disasters to evaluate the performance of structural models of hospital demand using these two approaches. Because these disasters severely damaged or destroyed hospitals but left the majority of the surrounding area undisturbed, they exogenously altered consumers' choice sets. This provides a unique environment to test these models since most choice environments for hospital demand are relatively constant over time and the changes that do occur may be endogenous to the policies being evaluated.

To test the stability of the structural demand parameters, we estimate prominently used models of hospital demand on data both before and after the hospital closures. In other words, we estimate the same models on a similar set of patients using two different choice sets, one with the destroyed hospital and one without it. With the estimates from these models, we will measure the stability of parameters by estimating the difference in implied counterfactual predictions for two policies: the effect of narrow network policies on insurance demand and the price effects of hospital mergers.

To test the predicted accuracy of the models for policy counterfactuals, we study the implied diversion ratios. Diversion ratios are commonly used to evaluate hospital mergers, and in this context, refer to the share of patients leaving the removed hospital who go to the other hospitals in the area. Using the model estimated in the period prior to the disaster, we predict the share of patients in the post period going to the other hospitals in the region. Since we observe this in the data after the hospital closure, we can directly check which models generate the most accurate predictions using this metric.