Understand the Real Reasons Reproducibility Reform Fails

Nature ^ | 06 December 2021 | Nicole C. Nelson

[nickcarraway]

Lack of rigour is often blamed on pressure to publish. But ethnographers can find out what truly keeps science from upping its game.

A decade ago, the US National Institute of Neurological Disorders and Stroke convened a workshop on how to improve the rigour of preclinical research. Its recommendations were surprisingly straightforward: scientists should mask (or ‘blind’) their studies; randomize; estimate appropriate sample sizes; and specify rules for data handling (S. C. Landis et al. Nature 490, 187–191; 2012). Ten years on, many preclinical scientists still do not take these basic steps.

Ask most advocates of rigorous science why this is, and they will answer with two words: perverse incentives. Scientists are rewarded for getting things published, not for getting things right, and so they tend to favour speed and ease over robustness. But as an ethnographer, this explanation has never sat well with me. I’ve spent more than 15 years studying biomedical research cultures, and scientists’ behaviours are rarely so transactional. So I decided to knock on a few doors at my institution, the University of Wisconsin–Madison, to ask researchers who work with animals why they were using the methods they were.

They explained that their decisions were based not on publication pressures, but on maintaining the integrity of experiments and respecting facility routines. For example, techniques to mask which treatment groups animals are in risk causing misidentification or cross-infection.

What NASA missions can teach us about teamwork

One scientist told me how years of tedious mouse breeding had gone off the rails when animals’ ear tags fell out, cage cards were swapped or spreadsheets had errors. The risk of mix-ups caused by masking seemed too great, especially in experiments where effects were so pronounced that there was no risk of bias.

[nickcarraway]

Let me know if anybody wants on a Replication Crisis ping.

[ProtectOurFreedom]

The biggest perverse incentive of all is the revolving door from the FDA to Big Pharma executive suites.

[SunkenCiv]

Scientists are rewarded for getting things published, not for getting things right, and so they tend to favour speed and ease over robustness... One scientist told me how years of tedious mouse breeding had gone off the rails when animals’ ear tags fell out, cage cards were swapped or spreadsheets had errors. The risk of mix-ups caused by masking seemed too great, especially in experiments where effects were so pronounced that there was no risk of bias.

• Stuffed Dormice A Roman Favourite [07/21/2003]
• Dormice, sea urchins and fresh figs: the Roman diet revealed [06/14/2011]
• Dinner At Piso's: Ancient Romans ate meals most Americans would recognize [02/07/2015]
• Diets of the middle and lower class in Pompeii revealed
• Romans went to war on diet of pizza, dig shows. [08/26/2002][01/05/2014]
• Roman gladiators' diet probably not what you'd expect, researchers find [10/21/2014]
• It Is Nuts To Serve Squirrel Burger (Andrew Zimmerman Weird Foods Alert) [10/04/2014]
• Fermenting herring prompted gas leak call [09/09/2012]

[sitetest]

I didn’t know there was such a list. You can add me.

[nickcarraway]

I am just starting it.

[sitetest]

I guess that makes me a charter member, LOL.

[Little Ray]

Me, too!