Posted on 03/21/2010 9:37:39 AM PDT by Ernest_at_the_Beach
The quote in the headline is direct from this article in Science News for which I’ve posted an excerpt below. I found this article interesting for two reasons. 1- It challenges use of statistical methods that have come into question in climate science recently, such as Mann’s tree ring proxy hockey stick and the Steig et al statistical assertion that Antarctica is warming. 2- It pulls no punches in pointing out an over-reliance on statistical methods can produce competing results from the same base data. Skeptics might ponder this famous quote:
“If your experiment needs statistics, you ought to have done a better experiment.” – Lord Ernest Rutherford
There are many more interesting quotes about statistics here.
- Anthony
UPDATE: Luboš Motl has a rebuttal also worth reading here. I should make it clear that my position is not that we should discard statistics, but that we shouldn’t over-rely on them to tease out signals that are so weak they may or may not be significant. Nature leaves plenty of tracks, and as Lord Rutherford points out better experiments make those tracks clear. – A
==================================
Odds Are, It’s Wrong – Science fails to face the shortcomings of statistics
March 27th, 2010; Vol.177 #7 (p. 26)
P valueA P value is the probability of an observed (or more extreme) result arising only from chance. S. Goodman, adapted by A. Nandy
For better or for worse, science has long been married to mathematics. Generally it has been for the better. Especially since the days of Galileo and Newton, math has nurtured science. Rigorous mathematical methods have secured science’s fidelity to fact and conferred a timeless reliability to its findings.
During the past century, though, a mutant form of math has deflected science’s heart from the modes of calculation that had long served so faithfully. Science was seduced by statistics, the math rooted in the same principles that guarantee profits for Las Vegas casinos. Supposedly, the proper use of statistics makes relying on scientific results a safe bet. But in practice, widespread misuse of statistical methods makes science more like a crapshoot.
It’s science’s dirtiest secret: The “scientific method” of testing hypotheses by statistical analysis stands on a flimsy foundation. Statistical tests are supposed to guide scientists in judging whether an experimental result reflects some real effect or is merely a random fluke, but the standard methods mix mutually inconsistent philosophies and offer no meaningful basis for making such decisions. Even when performed correctly, statistical tests are widely misunderstood and frequently misinterpreted. As a result, countless conclusions in the scientific literature are erroneous, and tests of medical dangers or treatments are often contradictory and confusing.
Replicating a result helps establish its validity more securely, but the common tactic of combining numerous studies into one analysis, while sound in principle, is seldom conducted properly in practice.
Experts in the math of probability and statistics are well aware of these problems and have for decades expressed concern about them in major journals. Over the years, hundreds of published papers have warned that science’s love affair with statistics has spawned countless illegitimate findings. In fact, if you believe what you read in the scientific literature, you shouldn’t believe what you read in the scientific literature.
“There is increasing concern,” declared epidemiologist John Ioannidis in a highly cited 2005 paper in PLoS Medicine, “that in modern research, false findings may be the majority or even the vast majority of published research claims.”
Ioannidis claimed to prove that more than half of published findings are false, but his analysis came under fire for statistical shortcomings of its own. “It may be true, but he didn’t prove it,” says biostatistician Steven Goodman of the Johns Hopkins University School of Public Health. On the other hand, says Goodman, the basic message stands. “There are more false claims made in the medical literature than anybody appreciates,” he says. “There’s no question about that.”
Nobody contends that all of science is wrong, or that it hasn’t compiled an impressive array of truths about the natural world. Still, any single scientific study alone is quite likely to be incorrect, thanks largely to the fact that the standard statistical system for drawing conclusions is, in essence, illogical. “A lot of scientists don’t understand statistics,” says Goodman. “And they don’t understand statistics because the statistics don’t make sense.”
====================================
Read much more of this story here at Science News
Statistics is strange....mathematics....usually.
Lies, damned lies and statistics...
I have to dig up an old text I used to teach a small section of statistics.
Was always unconfortable with it.
If a large number of people have come to intuitively understand that after reading decades of hysterical “science” news stories, then this isn’t really that much of a revelation. There are a lot of fields of study that are treated as a science when they’re not.
Brings to memory Bayes Theorem.
Statistics is a well understood mathematic discipline. It’s weaknesses and strengths are well documented. The problem is not the math behind science, it is the persons that are intentionally trying to sell a falsehood.
There are entire books written on how to deal with alpha and beta errors which this article insinuates are not understood.
Easy way to frame this argument. If you were a high school science student and had to use statistical “smoothing” to produce the “right” data, what kind of grade would you get?
If you do 20 tests to the 95% significance, one would be expected to report as true (null hypothesis rejected) even when the null hypothesis is true.
That is why a single test is meaningless, and can be manipulated. When different experimenters repeat the test, and find the same result, you then can have confidence.
Nothing is settled, ever. Even if the value of Pi started reporting differently (due to a change in the nature of the cosmos) then the current value of Pi would be studied, and the change reported.
Science doesn’t tell you what is true. It is a good way of finding out what is not true.
It seems to me that it is less over-reliance on statistics, than misapplication of statistics that is at fault in the state of “climate science”. And this misapplication has two aspects: the much-discussed one involving dishonesty—selecting weather stations about which statistical inferences must be drawn (due to gaps or the need to estimate an urban heat-island effect) in preference for ones with long continuous track records in rural areas, omitting weather stations in colder regions (e.g. the Andes) and making inferences based on “nearby” stations in areas with radically different climate—and one involving an honest conceptual error.
The conceptual error lies behind the “weather is not climate” mantra, that hides the fact that climate IS weather, averaged over longish-time intervals, but I think also hides the mistaken assumption that the variability of weather is random noise of the sort statistical methods are useful for dealing with. In fact, the unpredictability of weather is due to the underlying non-linear dynamics that does not go away when you take time averages.
Truth!
The article is, ironically, misleading. The problem is the misuse or misunderstanding of statistics, not that the scientific method is incorrect or that statistics is incorrect.
Another issue (that I will soon write about in my blog http://libertyphysics.wordpress.com/) is that statistical correlations are very often confused with cause and effect. For example, say a statistically correct study is done showing people in countries who eat more yogurt live longer than people in countries who eat less yogurt. I'm being simplistic, of course.
That doesn't mean that eating yogurt will make you live longer, no matter how correct the statistics. The scientists who do these studies usually know better but the fawning news media does not and reports such findings as if they were a call to action.
Worse, even if the statistics is done correctly, say we discover that, on average, eating salt raises a population's blood pressure, that conclusion doesn't necessarily apply to any given individual member of the population. For example, statistically, men are stronger than women. But I can easily find a woman stronger than the average man just as I can easilly find someone who can eat salt without an increase in blood pressure.
Misuse and misunderstanding is the issue.
We live in a very strange world in so many ways...
|
|
|||
Gods |
Thanks Ernest_at_the_Beach. |
||
|
· Discover · Nat Geographic · Texas AM Anthro News · Yahoo Anthro & Archaeo · Google · · The Archaeology Channel · Excerpt, or Link only? · cgk's list of ping lists · |
|||
AGW is extremely robust with respect to data, all observations confirm it at the 100% confidence level.
Well, if you can pick and choose data, literally anything’s possible. Also see Ancel Keys.
The current trend is to model a dynamic system, and then calibrating the model to historic statistics by using hidden assumptions, (which can be really wrong.) Then the modelers feel confident in using the model to predict the future and to regulate the heck out of you.
Sciencianism. That’s what it’s become. State funded Sciencianism.
Just like the arts. Throw too much money at anything, make the money too available, and you get what you pay for.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.
