Investigating the reproducibility of the social and behavioural sciences

Abstract

Published claims should be reproducible, yielding the same result when applying the same analysis to the same data. We assessed reproducibility in a stratified random sample of 600 papers published from 2009 to 2018 in 62 journals spanning the social and behavioral sciences. Authors of 146 (24.3% [95% CI 21.1 - 27.9%]) papers made data available to assess reproducibility, and for some others, we obtained source data to reconstruct the dataset. From 145 papers assessed, 76.3 (52.6% [95% CI 44.7 - 59.9%]) papers were rated as precisely reproducible and 104.5 (72.1% [95% CI 64.5 - 79.0%]) papers as at least approximately reproducible (within 15% of the original effects or within .05 of original p-values) after inverse weighting each of the 553 claims by the number of claims per paper. We observed higher reproducibility for papers from Political Science and Economics than other disciplines, for more recent than older papers, and for papers from journals that required data sharing.

Publication
forthcoming, Nature

Related