Replication Studies, An Inherent Good?

Scientific and medical research is funded and conducted with the intent to generate actionable evidence. Across a number of fields, the limited reproducibility of results – in particular those results which influence beliefs and policy actions – has emerged as a source of concern. Even in highly controlled pre-clinical and clinical studies, researchers have failed to replicate 89% (47 out of 53) of study results (Begley and Ellis 2012). The response to these alarming findings has been greater advocacy, funding, and research devoted to replication and confirmatory trials. What is lacking from the discussion is a consideration of the opportunity costs associated with diverting more research resources to replication studies. Replication is often treated as an inherent good, a central tenant of the scientific method. We aim to ask: when might replication studies be inefficient (vs. efficient) investments?

This project aims to quantitatively estimate the value of replication studies, which in various forms are common components of post-market regulation of new drugs. Such studies are particularly common among targeted therapies that are commercialized based on auxiliary (sometimes called “surrogate”) endpoints.

We argue that replication is not an inherent good; rather, replication and confirmatory studies are only worth the investment when the information gained has the potential to change decision making with benefits to society that outweigh the costs of performing the research. We apply decision science value-of-information methods to generate prescriptive evidence on when to conduct replication studies in both theory and practice. While the applied methods are broadly generalizable, we illustrate our message using the high-cost setting of new drug approval, based on auxiliary endpoints, where billions of dollars and millions of human lives are at stake each year.

We propose an empiric analysis of the U.S. Food and Drug Administration’s required confirmatory studies following “accelerated approval” of new drug-indication pairs to extend this research to a context with real-world regulatory policy implications. In doing so, we ask the question: do confirmatory post-approval studies add value in pharmaceutical regulation? Since its inception in 1992, the Accelerated Approval Program has represented a deviation from the typical pre-market burden of evidence generation for new medicines, a deviation which is intended to be remediated with post-market studies. Where post-market evidence is not adding value, or is generated in ways that are more costly than necessary, this work can provide critical information on how confirmatory study requirements fall short of their intended goals and what potential efficiency gains may be achieved under alternative regulatory requirements.