How to Lie with P-values

P-values are used in statistics and scientific publications, much less so in machine learning applications where re-sampling techniques are favored and easy to implement today thanks to modern computing power. In some sense, p-values are a relic from old times, when computing power was limited and mathematical / theoretical formulas were favored and easier to deal with than lengthy computations.

Recently, p-values have been criticized and even banned by some journals, because they are used by researchers, who cherry-pick observations and repeat experiments until they obtain a p-value worth publishing to obtain grant money, get tenure, or for political reasons. Even the American Statistical Association wrote a long article about why to avoid p-values, and what you should do instead: see here. For data scientists, obvious alternatives include re-sampling techniques: see here and here. One advantage is that they are model-independent, data-driven, and easy to understand.

Here we explain how the manipulation and treachery works, using a simple simulated data set consisting of purely random, non-correlated observations. Using p-values, you can tell anything you want about the data, even the fact that the features are highly correlated, when they are not. The data set consists of 16 variables and 30 observations, generated using the RAND function in Excel. You can download the spreadsheet here.

In some sense, p-values are a relic from old times, when computing power was limited and mathematical / theoretical formulas were favored and easier to deal with than lengthy computations.

And THAT, is where the writer surrenders his position.

As stated, correctly, there is nothing wrong with p-values...just like there is nothing wrong with guns. It is when p-values are put in the hands of unscrupulous people with the intent to do harm, that bad things can happen. Furthermore, models that have other problems like multicollinearity, can lead to inefficient parameter estimates and problematic p-values.

However, this is all Stat 101 stuff. The REAL intent behind this article, is to subtly paint statistics in a bad light and puff up the data science/machine learning professionals. Which is sad, since Dr. Granville has good credentials.

As others have said, there’s absolutely nothing wrong with p values. They are a tool for reporting the significance of an experimental result, but like most tools, they can be misused. This is especially true if researchers don’t really understand what they are and how to use them as seems to be the case in many areas of research.

For the unfamiliar, suppose you are conducting an experiment such as a drug trial. You would split your subjects into two groups, a control group that does not receive the drug and a treatment group that does. You then compare the two groups on some measure to see if that measure is different between the groups. All that sounds simple enough, but the problem is that even if you’d compare two groups receiving no treatment, they’d never give EXACTLY the same measurements. There would always be some differences. The big question then is how different do the results have to be before we can really claim that the drug did something.

That’s where p values come in. A p value is a measure of probability (hence the “p”) that tells you how likely it is that sampling two groups of people at random and performing the measurement on those random groups would give a difference at least as large as the difference that you observed between your control and treatment groups. If that probability is relatively large, then you probably haven’t observed a real difference. You would be likely to observe a similar difference between ANY two groups of subjects, even absent any treatment. If that probability is small then MAYBE you found something.

The value generally accepted in most scientific work is p=0.05. That is results are not considered significant unless there’s less than a 5% chance that they could have occurred randomly. That should be considered to be a guideline, but too often it seems to be a set in stone benchmark instead. There are many instances where p<=0.05 is completely inappropriate. As a simple example, suppose instead of one drug, you are testing 20 drugs as potential treatments for some condition. If you blindly use p<=0.05 as your cutoff, you would very likely be publishing a positive result when all you really found was a random difference. That should be obvious: if you roll a 20-sided die 20 times you wouldn’t be all that shocked if you rolled a 1 on one of your rolls. You wouldn’t conclude your die was somehow loaded based on that observation.

Similarly you should NOT automatically just publish a positive result in such a situation. A much lower p value should be used in such a case. Particle physicists, for example do just this when they search for new particles. They recognize that they are looking for a particle at a wide range of energies, essentially conducting a large number of experiments simultaneously. Therefore they require a p value on the order of 10^-9 in order to announce a result.

Speaking of smaller does for older people...

An old man walks into a pharmacy and hands the pharmacist a script.

The pharmacists says “I can fill this right now, Mr. Smith, you can wait right there.” as he points to a chair.

The old man (OM) says, “Hang on, Sonny, can I ask you a question?”

Pharm: “Sure Mr. Smith, what is it?”

OM: “Can you cut the pills in half?”

Pham: “Yes, I can do that.”

OM: “How about a quarter, sonny, can you cut them into quarters?”

Pham: “Well, Mr. Smith, this prescription is for Viagra. I’m afraid a quarter of a Viagra tablet will not give you a sufficient erection for sex, Sir.”

OM: “Sex? Erection, schmerection! I just want to stop peeing on my slippers!”

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.