What Is the Replication Crisis?

The replication crisis is an ever-present issue in scientific research. This article will look at this subject, providing a brief overview of this complex subject.

The replication crisis, also known as the reproducibility crisis and the replicability crisis, is a crisis that impacts the methodology of scientific research. Over time, it has been realized by several bodies that the results of many scientific studies are hard or almost impossible to accurately reproduce. The reproducibility of empirical data is essential for the scientific method, so difficulties in reproducing the results of a study or theory undermine its credibility.

Because of this, valuable scientific knowledge may be lost. The peer-review process is an integral part of scientific research, and if the data cannot be reproduced it cannot be validated. The crisis particularly affects the medical and social sciences. Considerable efforts in these fields have been performed into revisiting classic results, confirming their viability, and assessing the reasons for their failure if they are found to be unreliable. There is strong evidence from survey data that the crisis impacts every natural science field.

Growing awareness of the problem led to the term being coined in the early 2010s. The identification of the issue and considerations of causes and remedies have led to the new scientific discipline of metascience which uses empirical research methods to examine empirical research practices.

Both obtaining and analyzing data are involved in empirical research. Because of this, there are two categories of consideration concerning reproducibility. Reproducibility is the validation of the analysis and the interpretation of the data. Repeating an experiment or study to obtain new independent data which verifies the results of the original study is called replication.

Causes There are a few causes of low reproducibility. A major cause is publication bias, where studies can become statistically skewed in the chase for significant results and overwhelming the correct results. Additional causes include questionable data analysis practices such as researcher degrees of freedom, data dredging, and HARKing. There can also be a failure to adhere to good scientific practice due to the generation of unprecedented rates of new publications and data, and the desire to publish or perish.

Scope The journal Nature highlighted the scope of the issue in 2016 with a poll of 1,500 scientists. 70% of respondents reported that they had failed to reproduce the results of at least one of their peer’s studies. 87% of chemists, 69% of physicists and engineers, 77% of biologists, 64% of environmental and earth scientists, 67% of medical researchers, and 62% of all other respondents reported this issue. 50% had failed to reproduce one of their own experiments.

Additionally, several respondents reported that academic editors and reviewers told them to tone down comparisons to original studies when they reported failed replications. Sometimes the problem appears to be deliberate, as well – out of respondents to a 2009 study, 2% admitted that they had falsified studies at least once, and 14% said they knew someone who did. According to one study, this misconduct is more prevalent in medical research.

A 2021 study found that papers with reproducible results tend to be cited less than papers with findings that cannot be replicated in leading journals. The authors of the study stated that one of the reasons for this is that the review team may face a trade-off between maintaining standards and how interesting the paper is. Additionally, the authors stated that some papers create hype, and there is pressure to receive grants and court media coverage.

Psychology: A Field at the Center of the Controversy Related Stories Researchers conduct SARS-CoV-2 risk assessment with regard to wildlife research, rehabilitation, and control What does the Future of Microfluidics within Research Look Like? Research highlights importance of booster vaccine doses to protect against all Omicron variants Psychology has a particular problem with the reproducibility of study data and results. A 2018 meta-analysis study of 200 papers concluded that psychological research, on average, is littered with examples of low statistical robustness. Social psychology has been particularly implicated, but the fields of educational research, clinical psychology, and developmental psychology all have problems with reproducibility.

A study published in Nature Human Behavior failed to replicate 13 out of 21 behavioral and social science papers published in Science and Nature, two leading journals. In another study, 186 researchers from 60 laboratories failed to replicate 14 out of 28 findings despite large sample sizes. One explanation is the prevalence of questionable research practices in the field of psychology, including selective reporting, partial publication of data, post-hoc storytelling, and optional stopping.

In a study, a majority of 2,000 psychologists who responded admitted to using at least one questionable research practice. Solutions are difficult to realize in the field, but large-scale collaboration between researchers to make their data freely accessible is becoming more common practice.

Non-reproducibility of results in the field of medical science can have serious consequences. The US Food and Drug Administration, for example, discovered that 10-20% of medical studies between the years 1977 and 1990 were flawed. In 2012, Begley and Ellis discovered that out of 53 pre-clinical cancer studies, only 11% could be successfully replicated.

A survey by Nature discovered that more than 70% of respondents had failed to reproduce the results of at least one other study. Over half could not replicate at least one of their experiments. John Ioannidis of Stanford University has called for widespread reform, with one solution being to build a more patient-centric model of medicine instead of the current practices which favor the needs of physicians, sponsors, and investigators.

For research to be valid, the results and data must be reproducible, and currently, there is a crisis in multiple scientific fields. There are no easy solutions to the replication crisis, but researchers, journals, academic editors, patients, and advocacy groups can play a role in ensuring that robust standards are adhered to in scientific research. With increasing focus on the replication crisis, there may be a way forward that will improve the validity and quality of research exponentially.

References: Serra-Garcia, M & Gneezy, U (2021) Nonreplicable publications are cited more than replicable ones Sci Adv 7(21) eabd1705 [online] Accessed 27th Jan 2022 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8139580/ Stanley, T.D. et al. (2018). What Meta-Analyses Reveal About the Replicability of Psychological Research Psychological Bulletin, 144, 1325–1346 [online] Accessed 27th Jan 2022. https://pubmed.ncbi.nlm.nih.gov/30321017/ Camerer, C.F et al. (2018) Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015 Nature Human Behavior 2 pp. 637-644 [online] Accessed 27th Jan 2022 https://www.nature.com/articles/s41562-018-0399-z/ Ionnaidia, J.P.A (2016) Why Most Clinical Research is Not Useful PLoS Med 1396) e1002049 [online] Accessed 27th Jan 2022 https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1002049

Good article.

This is why almost all of “social science” is ^%$, btw.

It is virtually impossible to go back in time and replicate initial conditions so almost all claims are impossible to verify using the scientific method.

It is virtually impossible to go back in time and replicate initial conditions so almost all claims are impossible to verify using the scientific method.

I don't mind if someone wants to believe in Evolution, or in Climate Change. But, to me, these are religious issues that are dogma for the science community, but which should not really be considered Science by anyone. Nice ideas, comforting beliefs, if you are a certain kind of person.

“John Ioannidis of Stanford University”

He’s famous for this eye opening paper.

Why most published research findings are false
https://pubmed.ncbi.nlm.nih.gov/16060722/

And caught hell from the establishment for publishing this.

John Ioannidis: COVID Deaths Massively Over-counted
https://covidcalltohumanity.org/2021/09/30/john-ioannidis-covid-deaths-massively-over-counted/

If you can not reproduce the orginal data the first data is void. Or as my wise grandfather said, “It is Bull S—t.

Many of our professional journals will publish non reproduced conclusions of a theory based on politics and not science. Thus they are purveyors of intentional disinformation.

When I was in geology and later pharmacy if a grad student did such he would have been shown the door. Sadly this is not done today.

1. Theory
2. Prove the theory
3. Repeat again to make sure you can reproduce the proof of the theory
4. Have peers repeat it and if the same results publish
5. Once published others will also repeat the study
6. If they come up with the same results it is very probably valid.

The vast majority of our past scientific journals of great, are no more. If it is pure science with no political gain it is pretty good. If the subject has political ramifications, just stop reading it. It will not be true.

Scientific America
National Geographic (their maps are even political)
Britain’s Royal Society
All science programs from the media particularly BBC
and many others have committed suicide as professional journals.

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.