Over the past few years, there has been a growing awareness that many experimentally established “facts” don’t seem to hold up to repeated investigation. This was highlighted in a 2010 article in the New Yorker entitled The Truth Wears Off and since then, there have been many popular press accounts of different aspects of science’s current reproducibility crisis.These include an exposé of the increasing number of retractions by scientific journals and damning demonstrations of failures to replicate high profile studies.
Articles in recent days have discussed how the majority of scientists might be more interested in funding and fame than “truth” and are becoming increasingly reluctant to share unpublished details of their work. So why exactly is science in such a crisis – and where do we start fixing it? What caused the reproducibility crisis?
In each discipline, there have been different triggers. In psychology, it was an unreplicable study about extrasensory perception (ESP); in medicine, it was unreplicable cancer studies. Behind these (somewhat arbitrary) triggers, however, are the same underlying causes: a combination of mechanised reporting of statistical results and publication bias towards “statistically significant” results. Problems with traditional significance testing and publication bias have already been addressed on The Conversation.
So is the crisis a result of scientific fraud? Not really. Well, maybe a bit. The number of known cases of outright fraud is very low. But what we might consider softer fraud—or “undisclosed flexibility” in data collection—is well documented and appears to be very widespread. … such as this one: Open Science Framework.
There can be little doubt that the “publish or perish” research environment fuels this fire. Funding bodies and academic journals that value “novelty” over replication deserve blame too. While no-one knows the true level of undetected scientific fraud, the best way to deal with this problem is to increase the number of replication studies.
How do we fix it? Some initiatives are already underway. In psychology, there’s the Reproducibility Project, which has previously been covered by The Conversation.
In biomedicine, there’s the Reproducibility Initiative. It’s backed by the Science Exchange, the journal PLOS ONE, Figshare, and Mendeley. It will initially be accepting 40 to 50 studies for replication with the results of the studies to be published in PLOS ONE.
There are also various other proposals such as
- a “reproducibility index” for journals, similar to an impact factor
- changes to the regulations of funding bodies
- random audits.
The proposals and initiatives mentioned above draw attention to improving methodological protocols, and require a more thoughtful approach to statistical reporting practises. Via Science is in a reproducibility crisis: How do we resolve it?.