The State of the Statistical Science for Evaluating Opioid Policies: Findings from Simulations

The nation is in the midst of an opioid-related public health crisis. Drug overdose fatalities in the U.S. today, driven by prescription and illicit opioids, far exceed those from any prior drug epidemic on record and even exceeds the number of deaths associated with the HIV epidemic of the 1980s (Humphreys, 2017). The upward trends in mortality over the past two decades have affected both men and women, and no age group or geographic region of the U.S. has been immune [Ruhm, 2017; Hedegaard et al. 2017]. States and the federal government have tried a plethora of strategies aimed at curbing the tide of the supply of opioids, the misuse of them, and the mortality associated with their misuse, which has produced a state policy landscape that is both complex and dynamic. This creates at least three specific challenges for those interested in evaluating the impacts of these policies. First, there are issues of policy endogeneity, yielding low quality “control” groups for causal inference. Second, implementation of a given policy in one state might differ substantially from implementation in another state. This reduces the strength of the signal provided when heterogeneous polices are pooled together and evaluated as if they are the same. Finally, the simultaneous adoption of multiple policies together, rather than singular policies in isolation, can generate differential policy effects. For instance, policy refinements in one area (making a PDMP “must access” for prescribers) while also implementing policy in another area (e.g. co-prescribing of naloxone) may make one policy appear to be more effective than it actually is. RAND’s Opioid Policy Tools and Information Center (OPTIC) of Excellence attempts to educate researchers and policy makers of these issues, providing standardized information relevant for conducting policy evaluations, and educating users of opioid policy data and methods on how to generate better policy analyses with robust methods for better inference.

In this paper, we provide an overview of the state of the statistical science in the opioid policy space and describe simulation findings regarding the statistical properties of commonly used methods. We begin by discussing findings from an extensive literature review on opioid policy including a summary of the types of causal inference methods currently being used in practice. We then introduce findings from a simulation study that aims to empirically identify the best model or model(s) to use for three commonly used opioid related outcomes (state per capita prescription opioids prescribed, state per capita opioids distributed, and opioid related mortality). The simulation assesses the statistical properties (e.g., robustness, power, type I error rates) of commonly used regression models for estimating the effectiveness of opioid policies with the goal of providing insights into the limitations and sources of bias introduced by existing methods. Using simulation studies to assess the performance of both commonly used and newly developed statistical methods in opioid research is a powerful innovation that can yield accurate inferences on which policy decisions can be based.