How to Lie with P-values

Data Science Central ^ | 6-11-19 | Vincent Granville

[spintreebob]

P-values are used in statistics and scientific publications, much less so in machine learning applications where re-sampling techniques are favored and easy to implement today thanks to modern computing power. In some sense, p-values are a relic from old times, when computing power was limited and mathematical / theoretical formulas were favored and easier to deal with than lengthy computations.

Recently, p-values have been criticized and even banned by some journals, because they are used by researchers, who cherry-pick observations and repeat experiments until they obtain a p-value worth publishing to obtain grant money, get tenure, or for political reasons. Even the American Statistical Association wrote a long article about why to avoid p-values, and what you should do instead: see here. For data scientists, obvious alternatives include re-sampling techniques: see here and here. One advantage is that they are model-independent, data-driven, and easy to understand.

Here we explain how the manipulation and treachery works, using a simple simulated data set consisting of purely random, non-correlated observations. Using p-values, you can tell anything you want about the data, even the fact that the features are highly correlated, when they are not. The data set consists of 16 variables and 30 observations, generated using the RAND function in Excel. You can download the spreadsheet here.

[spintreebob]

the fact that the features are highly correlated, when they are not

Careful what we believe.

[SanchoP]

Long before computers there were lies,dam’lies and statistics.

[Repeal The 17th]

My next-door-neighbor and I were discussing this very thing
a few nights ago at my fire-pit with some good bourbon.

[IncPen]

I contend that if the climate ‘researchers’ had to testify as to their grant money and their findings under oath the entire hoax would collapse

[I want the USA back]

Since I don’t know what a p-value is, I cannot appreciate the depth of wisdom shown in the article.

[Robert DeLong]

And yet with computers the charade continues vis-a-vis corrupt code. 8>)

[Paradox]

Statistics done Lie, but Liars use Statistics.

[heavy metal]

the trans community has been lying about p values for years...

[cpdiii]

“Recently, p-values have been criticized and even banned by some journals, because they are used by researchers, who cherry-pick observations and repeat experiments until they obtain a p-value worth publishing to obtain grant money, get tenure, or for political reasons”

There is nothing wrong with P Values. If you cherry pick the data all the results are crap. It is no longer valid data.

[rhombus10]

95% of an unknown number of scientists agree with me./s

[freedumb2003]

>>Long before computers there were lies,dam’lies and statistics.<<

Computers have the ability to make many errors very quickly.

[BipolarBob]

And here I thought this was going to be helpful when I met with my parole officer. Aaaarrgghh!

[the_Watchman]

Since we are talking about p-values, it would have been a lot more believable if you had stated that you were drinking beer.

[Repeal The 17th]

You are right!

[bigbob]

Yep, looong before:

[sauropod]

There are liars, damn liars, and statisticians.

[Paladin2]

This is the kernel of how Climate “Science” works.

[MeganC]

Statistically speaking, people who drink bourbon near fire pits at night are happier than those who do not.

(-:

[Repeal The 17th]

Will you marry me?

[LambSlave]

Recently, p-values have been criticized and even banned by some journals, because they are used by researchers, who cherry-pick observations and repeat experiments until they obtain a p-value worth publishing to obtain grant money, get tenure, or for political reasons.

Also, journals often refuse to consider publishing results with negative findings, which makes the problem much worse.

For example, Researcher A repeats experiment X with slight (but insubstantial) modifications, and finds that his sugar water kills cancer cells better than chance with p<0.05. Researchers B, C and D repeat the experiment but find it does nothing, and their department chair tells them to move on to the next series of experiments, because he believes (rightly) that journals are looking to publish positive findings, and generally findings of no effect will not make it through the peer review process (which is long and time consuming). I don't believe these things are intentional though. When I did biomed research I saw this a lot; researchers had good intentions but did not understand statistics. It is not just ignorance, it is often lack of sufficient intellect to be a good scientist. It would be nice if every researcher were a genius who had a natural, confident grasp of all scientific disciplines related to their area of study, but i think we set a pretty low bar for entry into STEM careers.