The laymen usually misunderstand how little “90%” is as a confidence level - and some traders with fear masterfully abuse this ignorance. 90% vs 10% is not that “qualitatively” far from 50% vs 50% - and one can transform one to the other by a “slight” pressure in the methodology and the formulae. If you want to be scientifically confident about a conclusion, you should really demand 99.9% or more. And it’s actually not that hard to obtain such stronger evidence assuming that your hypothesis is actually correct and the “signal” exists.
http://motls.blogspot.com/2010/03/defending-statistical-methods.html
In particular, discoveries of new particles by colliders do require 5 sigma. No one would have claimed a discovery of a top quark at 3 sigma - which would only be viewed as a suggestive yet vague hint.
Once again, this increase is needed because people often cook their results to make “discovery claims” that are bogus: it’s easy to “improve” the tests. If you try 10 variations of the same test, one of them will show a (fake) effect at a 90% confidence level: that’s what the 90% confidence level means, by definition. Unfortunately, many researchers are approaching the things in this way.
With a 5-sigma discovery, such cheating becomes virtually impossible because you would need a million of variations of your paper - and only one of them would show a fake positive. On the other hand, it’s not “infinitely more difficult” to get 5-sigma results relatively to 3-sigma results. Because the relative errors go like “1/sqrt(N)” where N is the number of events (whose average you’re calculating, in a way), you only need to increase the number of events by a factor of “(5/3)^2 = 2.7778” to go from 3 sigma to 5 sigma.
Thanks AdmSmith.