Interesting article
Revelation 4:11Intelligent Design
See my profile for info
Posted on 08/30/2005 10:29:44 AM PDT by LibWhacker
Most published scientific research papers are wrong, according to a new analysis. Assuming that the new paper is itself correct, problems with experimental and statistical methods mean that there is less than a 50% chance that the results of any randomly chosen scientific paper are true.
John Ioannidis, an epidemiologist at the University of Ioannina School of Medicine in Greece, says that small sample sizes, poor study design, researcher bias, and selective reporting and other problems combine to make most research findings false. But even large, well-designed studies are not always right, meaning that scientists and the public have to be wary of reported findings.
"We should accept that most research findings will be refuted. Some will be replicated and validated. The replication process is more important than the first discovery," Ioannidis says.
In the paper, Ioannidis does not show that any particular findings are false. Instead, he shows statistically how the many obstacles to getting research findings right combine to make most published research wrong.
Massaged conclusions
Traditionally a study is said to be "statistically significant" if the odds are only 1 in 20 that the result could be pure chance. But in a complicated field where there are many potential hypotheses to sift through - such as whether a particular gene influences a particular disease - it is easy to reach false conclusions using this standard. If you test 20 false hypotheses, one of them is likely to show up as true, on average.
Odds get even worse for studies that are too small, studies that find small effects (for example, a drug that works for only 10% of patients), or studies where the protocol and endpoints are poorly defined, allowing researchers to massage their conclusions after the fact.
Surprisingly, Ioannidis says another predictor of false findings is if a field is "hot", with many teams feeling pressure to beat the others to statistically significant findings.
But Solomon Snyder, senior editor at the Proceedings of the National Academy of Sciences, and a neuroscientist at Johns Hopkins Medical School in Baltimore, US, says most working scientists understand the limitations of published research.
"When I read the literature, I'm not reading it to find proof like a textbook. I'm reading to get ideas. So even if something is wrong with the paper, if they have the kernel of a novel idea, that's something to think about," he says.
Heck, I'd be happy if just the NIH would stop throwing taxpayer money at poorly conceived and executed studies.
"When I read the literature, I'm not reading it to find proof like a textbook. I'm reading to get ideas. So even if something is wrong with the paper, if they have the kernel of a novel idea, that's something to think about," he says.
Speaking as someone who has served his time in a research lab in graduate school, I can testify to the truth to this. As a general rule, nobody reads journals for any kind of scientific proof (occasionally a paper comes along that really does definitively prove something, but it's pretty rare). We read them to get ideas, and to gather a general feel for what other researchers are interested in.
Scientists are generally aware of one another's biases. When you read a scientific paper, it's usually pretty darn clear what the scientist was trying to show - what conclusion the scientist would have liked to see. If you read these papers as a trained scientist, you learn that their real value lies not in the researcher's outcomes, but in what the researcher had to go through in order to produce those results! That is how you actually learn anything about reality when reading scientific papers.
A good friend of mine used to work as a lab tech for animal experiments, or as he puts it, he "tortured animals for a living."
He tells the most appalling stories about how studies were designed to avoid finding what they didn't want to find.
The phenomenon you describe is unfortunately not at all limited to those with a leftist agenda. Money can also do a good job of motivating a scientist to find a way to find what he, or his backers, want found.
You should. The statistics they cite are unexceptional with respect to the multiple hypotheses problem (eg if you test 20 false hypotheses at the 95% confidence level), one of them will probably test as true at the 95% level simply as a matter of random noise.
For example, suppose you are testing the output of DNA chips as a preditor of diabetes. There are some 2,000 hypotheses per experiment. That is, the chip produces 2,000 measurements for, say, 2,000 genes, and then those measurements are correlated with a disease state. At the 95% confidence level, you are bound to get many many correlations that seem significant that are only noise.
I have done a lot of scientific publication, refereed journals and conferences, and serve as the editor of peer-reviewed journal. Many, many first published results are wrong. Not just because of the noise issue. Scientists are folks like everyone else and they will ignore the negative findings and report only the positive. They will not define their endpoint in advance but keep going when it looks like they are almost there and so on. These are statistical no no's but they happen all the time.
Some years ago, I found an error in a paper I had previously written with another author--we were measuring (and reported) a bug in our software, not a real phenomenon. He was the head of a prominent dept in a prominent university. He argued 'til he was blue in the face that we should not report that we had been mistaken.
The system tends to be self-correcting because other folks try to replicate the wrong results and cannot. That gets reported. So eventually the noise is filtered out.
The article says that most scientific articles are wrong. I don't know why anyone would find that controversial, or even surprising.
For creationism to hold sway, however, they'd essentially all have to be wrong.
WOW. The very pinnacle of bad science writing. What the headline really should say is that most experiment designs, even those resulting in peer-reviewed publications, are tendentious crap put together by sub-competent "scientists."
Revoking accreditations at some PhD programs seems in order.
This is statistics, not science! Science deals with absolute experimentally reproducible phenomena. If you drop something, it will accelerate downwards at the same rate everytime.
Medical and sociological research is NOT science. It may or may not include science in measuring phenomena and mathematics to analyzed observed data.
If 1 person in 100 will die from something it does not mean 100 will die 1%. It means 99 people will not die 100% of the time and 1 person will die 100% of the time.
There is a terrible lack of understanding of science and statistics in this world. 99.99% of the media never took advanced science and or actuarial degrees.
I believe that half the artictle is true!
Ioannina family BUMP!!!!!!
Seems very timely given the no pain abortion study,
EVERY SINGLE envirowacko we are doomed study,
EVERY cancer study,
EVERY genes cause X behavior study...
There is some truth in this. But the factor that keeps it in check is stuff like rivalry, jealously, and wanting to be the guy that knocks off someone else. Plus, a lot of scientists will get a notion that something is wrong and can't let go of it until they satisfy themselves.
Where it doesn't work (in the short term) is in the incredibly sloppy field of computer modeling (global warming being only one small example of sloppy work in that field). There is not usually much emphasis in commercial and academic modeling circles on tying the model to groundtruth. No peer-pressure and ridicule for being wrong means no emphasis on actually being right. An elegant model is cool and significant even if it is completely wrong.
Interestingly, in military modeling circles, the folks there are rabid about tying the models to reality. Maybe because people die when they do not.
I bet the false results are nearly 100% when it comes to environmental issues.
Why? Do all scientific articles have something to do with evolution? Surely not everything in physics, chemistry, etc.?
I'd wager that more than 99.99% of everyone has no advanced science or actuarial degrees.
I would think that money/acclaim does and always has driven research. Scientists are just like all the rest of us: seldom are our motives pure. There are, of course, saints: the men as driven by the desire for natural truth as mystics are for a vision of God.
One of the main reason that papers are published are that they are very likely to have some flaws, no matter how minor and need peer review.
Not to quibble, Bill, but a statistical distribution should be reproducible (95% of the time at the 95% confidence level). If, of course, it is measuring something real.
Interesting article
Revelation 4:11Intelligent Design
See my profile for info
I don't believe that....
dang...does that make you right or wrong???
8^)
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.