Are political polls accurate? CA recall study of 20 polls says...NEVER TRUST POLLS AGAIN!

RealClearPolitics.com & original material ^ | 10/21/03 | Wolfstar

[Grampa Dave]

There is another aspect of these polls that can't be studied.

I was retired during the 2000 elections. It was amazing how often I got calls from pollers during the time before the election.

Later when the data showed that GW would win (before the dead rat votes and illegal alien votes were counted), my interviews got really short.

It went something like this, "Are you a registered republican, democrat or independent?"

When I answered a registered Republican, the next question was often, "Would you describe yourself as a liberal, moderate or conservative Republican?"

After I answered "A conservative Republican." That was usually the end of the interview. I was never polled.

[ampat]

Lots of time, energy and money spent on polls and this post. All I can say is: DUH!

[alwaysconservative]

Oh, I agree with YOUR conclusions, not those of the media! Actually, the point I was trying to make (and apparently not doing a very good job of it!) was to question whether or not "push polls", which I consider to be invalid, should be included in this tally, since their purpose was NOT to measure a random voter response, but to push or telemarket a certain view. It was not a legitimate poll in the sense of what you are evaluating, even if it was accurate on a couple of points. It's the difference of "proving" something with anecdotal evidence or legitimate scientific methodology.

[DoctorZIn]

This post has been added to the… California In Transition- Must read Threads!

Want on our daily or major news ping lists? Freepmail DoctorZin

[Doctor Stochastic]

The article seems almost as inaccurate as the polls in some ways. For example, the yes-no vote are described as though they were independent variables rather than correlated. The MOE's are incorrectly described for multiple choice polls (although the polls themselves probably do just as badly.) Additionally, the MOE only describes the statistical error in the sample; polling (as pointed out by the pollsters) is subject to greater error through systematic effects such as question wording, etc.

[Congressman Billybob]

Back on August 12, in my article "Nuts and Bolts in California," I not only successfully predicted all the results of the California recall election, I also predicted the incompetent struggle to understand what happened, after the polls were proven grossly inaccurate. (In all fairness, I missed exactly one prediction. I wrote that there would be one extra Republican candidate whose vote totals would not effect the outcome of the election. But I thought that hoeless Republican would be Bill Simon, rather than Tom McClintock.) I am curious about the methodology of RealClearPolitics. The only three polls that were close to the final pattern of this election across the board were conducted by Stanford University. For some reason, the source claims they did only one. They did three. And the reason they were closer than all the others to the final result is that THEY USED THE ACTUAL CALIFORNIA BALLOT rather than phone questions, to gauge the intentions of the potential voters.

The Editors of RealClearPolitics are usually pretty sharp cookies. But on this unique election, with the evidence in their hands, they are still missing the boat. Big time.

Congressman Billybob

Latest column, "Three People who Have it Coming," discussion thread. IF YOU WANT A FREEPER IN CONGRESS, CLICK HERE.

[Saundra Duffy]

Thanks for a very interesting analysis. I'll pass it on. Thanks again.

[Pres Raygun]

Excellent work. For long time Freepers, the inaccuracy of polls is well known. The MOE of a poll is greatly misunderstood by the public. MOE refers only to the precision of the poll and has nothing to do with the poll's accuracy. Technically precision is a measure of how a particular polling methodology will give the same result if conducted over and over again. For instance if a polling company conducted the same poll three times on the same day, each of the three polls should give the same results within the MOE. Accuracy on the other hand refers to whether or not a particular measurement (poll) is correct or gets the right answer. Precision (MOE) has nothing to do with accuracy. For a poll to be accurate, the polling sample must be unbiased, meaning the people polled must be statistically representative of the actual electorate. Pollsters realize that they never have a truly unbiased sample. Consequently, they try to compensate for the bias inherent in their methodology, but fudging the raw poll results, by correction factors. The only way for a pollster to determine these correction factors is to do the same analysis you have done on their own polls.

In conclusion polls are not to be trusted. Polls with an MOE of 3% may be inaccurate by 20% and the public has no way of knowing the accuracy of a poll until after the election. Tracking polls have some limited value in showing trends, because movement from one candidate to another is somewhat independent of the sample bias and of the pollster's correction factors. Only the direction of the trend is somewhat reliable, since the magnitude of the trend is dependent on the sample bias and the pollster's correction factors. Unfortunately, trends are often only a few percentage points and therefore are under the polls MOE making them meaningless.

[Wolfstar]

Your personal experience is echoed in one way or another by many, many people. I think many of us are intuitively suspicious of polls, but we go along with them partly because we enjoy trying to predict the future, and partly because we're told they are "scientific."

Why is it that between the late 1700's and the mid 1900's, Americans managed to elect all sorts of people to all sorts of positions without polls? Yet these days, we seem more dependent on them than ever. We are like people using a crutch when it isn't necessary. Well, a good challenge every now and again never hurt any institution.

[Grampa Dave]

Phoney poll and/or skewed results have been a major tool for the Rats for over a generation.

The recall of Davis and election of Arnold pulled back the curtain on this tool re the LA Slimes and other fishwraps in California.

[Wolfstar]

You're doing fine. I enjoy our electronic conversations and hope you do to. Actually, I made no judgements about these 20 polls. I have no way of knowing if any were push polls or not. Just used the list from RealPolitics.com because it was easily available and accessible. And I took the pollsters at their word. Each of them said their result was accurate within a given MOE. So I took their numbers, compared them to the election outcome, looked at the MOE's and listed the results. Eitehr they were accurate or they were not. Not fancy, but a very straightforward, common man's test.

It's worth bearing in mind that this is a CASE study, not a broad, all-encompassing study. One of my concerns about polling is that few pollsters make their methodology public. That goes to the heart of your point about whether or not one is a push poll. Every single one of these organizations claimed to be accurate. As a member of the general public, I decided to use the tools available to me to test their claims.

[Wolfstar]

...yes-no vote are described as though they were independent variables rather than correlated.

A potential voter has three choices on any ballot issue: to vote yes, to vote no, or not to vote. They are independent choices, and ALL pollsters give separate percentages for yes and no. Because they give separate predictions, each prediction can be tested against the MOE.

The MOE's are incorrectly described for multiple choice polls

In what way? A pollster says he predicts a vote of 55% for yes and 45% for no, and then gives a range plus or minus those figures within which he can claim to be accurate. If the actual election result falls within his MOE, he's predicted the outcome correctly. If not, he's wrong. What's so complicated?

[Wolfstar]

Hello, "Congressman." I followed your posts on the recall threads, and you most definitely did predict the outcome correctly. I also did, but on gut political instinct based on many years of involvement in grass-roots Republican politics here in the Los Angeles area.

RealClearPolitics.com did not do this case study, I did. I used the list of polls on their site. They only listed one for Stanford. Please use the links provided to see for yourself. I took those 20 polls as a CASE and tested their accuracy.

I'm just a member of the general public who has no pretensions, but does have a desire to evaluate the quality of the product pollsters sell. After all, many politicians make vital policy decisions only after checking the latest polls.

[Revolting cat!]

Polls are accurate as far as they represent the answers provided by their respondents. Beyond that, they're bogus from the getgo, "scientific" explanations notwithstanding. 1000 whatever responses represent 1000 whatever responses, not millions, sorry!

People lie to others, to themselves, people don't know and will not admit it, or polls don't allow them, people feel pressure answering questions, people are vain, cynical, flippant, etc.

The only polls that matter are at the cash register (or the ballot box as the case may be.) It's been demonstrated many times that people lie, consciously or not, about the food they eat, about their driving habits, and so on. I used to participate in focus groups and one of the more interesting things I noted was that the participants expressed totally different, and seemingly more honest opinions in the elevator going down after the focus group session from those expressed during the session. Astrology works better and is more fun too!

[Wolfstar]

Thanks for your thoughtful reply. As already mentioned on the thread, as a member of the general public, I take an MOE to be exactly what the media and pollsters represent it to be — a range of percentage points within which a given poll result can be considered to be accurate. Your point that MOE's are greatly misunderstood by the public goes to heart of my premise: That the public has absolutely no way to verify the accuracy of most polls.

What I have done in a kind of big way is to reflect exactly what the general public, most in the media, and even pollsters, themselves, do after every election. Election after election, people routinely look at the last polls that came out, compare them to the actual election results, and make judgements about which polling organization was the most accurate based on whose poll percentages came the closest to the actuals. In other words, they do exactly as I have done in this case study. Except in this case, I looked at 20 polls instead of just a handful.

[Uncle Hal]

Wolfstar-Great post. You must have done a lot of reseach. I hope you can get paid in some way for it. A couple of observations. 1) I believe a reason most polls got McClintock right is because he was a true conservative and many of the voters on the right were loyal to him no matter what. 2) I was polled over the phone and was asked who I was going to vote for in the recall. When I said Arnold, she said "me too". She also said some others things that were pro recall when she got the drift that I was for it. I do not think I would put much faith in that poll, as far as accuracy is concerned.

[k2blader]

Really gotta read this later bump.

[MattAMiller]

Nope. Voter surveys conducted AFTER the election showed most voters made up their minds a month ago. How come NONE of the polls caught this? This is another area where they didn't see voter behavior coming.

Maybe the variation is accounted for by the minority which didn't make up their minds early on. Maybe the people who said they made up their minds months ago were lieing.

[Doctor Stochastic]

The press is pretty bad at representing the MOE. It is not necessarily a rating of accuracy or confidence of the polls. Usually, they press gives what's called a 95% confidence interval. Unfortunately, this doesn't mean that the results fall within the MOE 95% of the time either.

Actually, "yes," "no," and "abstain" cannot be independent. They must sum to 100% and thus at most two are independent.

In most polls, the systematic errors are still bigger than the statistical errors. The article did point out some of these problems.

[Doctor Stochastic]

Very simple. Very clear. Very straightforward.

But incorrect. This is not what the MOE means. Neither press nor the pollsters generally explain things very well.