Is Good Research on Drugs Worth It? The Effect of Research Methods on Research Conclusions and Diffusion

Randomized clinical trials are the gold standard treatment evaluation, as their design potentially reduces some biases of non-randomized studies. However, when they fail to reduce risks of bias appropriately, methodologically inferior randomized clinical trials can yield mistaken conclusions, inflated treatment effects or greater variability. Due to data and methodological challenges, empirical evidence on the relationship between research methods and research outcomes remains limited. This study utilizes a new large database of clinical trials on prescription drugs included in gold standard systematic reviews. The database permits the mapping of the full text of each article, methodological assessment of the studies, and the study conclusions, to detailed bibliometric and funding information. This paper estimates the effect of higher quality research methods on research conclusions and diffusion concerning six important outcomes: the probability of reaching a favorable conclusion; the magnitude of the treatment effect; the variability in results across similar trials; the probability of reaching statistical significance; the clinical trial “fragility index” (a number indicating how many patients would be required to convert a trial from being statistically significant to not significant (p ≥ 0.05)); and the diffusion of medical knowledge.