Teddibly beastly.
Posted on 07/13/2002 8:15:16 AM PDT by aculeus
Yes, a bunch of quacks in America have done a "statistical survey" of the effects of Hormone Replacement Therapy. The media uniformly and dutifully reported the whole ragbag. The Times sub-headings read:
US study halted after health fears rise Patients suffer 41% increase in stroke risk 22% increase in risk of heart disease
Regular readers will note Relative Risks of less than 2, which are not accepted by real science. Having a computer model to hand, or our one hundred imaginary professors if you like, it seemed too good an opportunity to miss. There was a slight modification to include the fact that the number of controls was the same as the number of victims.
Unlike climate models there is nothing secret about it and any one with MathCad or similar can reproduce it with ease.
Taking one set of figures (the most drastic) from The Times, for strokes, ignoring the usual odd fraction of women as the numbers never quite add up, we can deduce that there were 8,304 women who took the therapy and 8,304 in the control group. The annual probability of getting a stroke was 0.00195, so the annual number of strokes in the control group was about 16 and the number in the victim group was about 23. This is a RR of 1.44, but what is 0.03 between friends?
On the assumption that there is no real effect, so the control and victim groups are statistically identical, two sets of 100 random binomial numbers were generated (using for the cognoscenti rbinom(100,8305,0.00195)) and one set was divided by the other. The results are then Relative Risks for the case where there is no causal relationship.
The results in the form of a histogram, were as follows:
The remarkable thing is that our 100 imaginary professors, working on no real effect at all, have actually done as well on average as the real quacks. For, when we include the publication bias effect, as in the second histogram, the imaginary professors got an RR of 1.43.
Furthermore, this was not an isolated case. Running the model ten times produced RRs of 1.44, 1.51, 1.49, 1.53, 1.38, 1.65, 1.41, 1.63, 1.40 and 1.53.
The question arises as to why these results are more dramatic than in the previous modelling exercise above. The reason is that this time we have included the effect of the scatter of the control group, which is the denominator in the RR. Thus, while both groups produce the expected relative scatter (standard deviation/mean) of 0.25 (i.e. the reciprocal of the square root of the expected number, 16), when we divide one by the other the relative scatter is increased by more than half.
So all this panic is caused by experimental results that are just about exactly what you would expect if there were no causal effect at all .
Overcome by the "significance" of their findings, the "researchers" called a premature halt to the study and there are now proposals that this (presumably) valuable therapy will be withdrawn. The shares of the company producing the drug fell to a two year low.
Another aspect of this scare is a new variation of publication bias. The Times chose to ignore two apparently beneficial (though equally fatuous) effects of this data dredge, as good news is no news.
Footnote for mathematicians: obviously an analytical solution to this problem would be preferable. I have tried and failed. It was beyond the mathematics of a simple engineer. I am aware that in the case of normal variates the quotient follows the rather beastly Cauchy density function, but I cannot work out what it should be for these asymmetric distributions. Any advice would be appreciated.
Correspondence
Teddibly beastly.
Math is hard.
No, really! I'm as unmathematical as they come.
FWIW, here's the bottom line (which is, more than anything, related to the "bottom line"): (1) Scary news sells better than anything; (2) People want things condensed for them so they don't have to think. This is even true of corporate CEOs and whatnot. (3) All this leads to the opportunity for GROSS mischaracterizations like this.
Stossel mentioned that after all the hoopla surrounding the breast implant flap, there is still not one, single shred of evidence that those things caused illness. Tell that to Dow.
What he said.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.