Are political polls accurate? CA recall study of 20 polls says...NEVER TRUST POLLS AGAIN!

RealClearPolitics.com & original material ^ | 10/21/03 | Wolfstar

[Wolfstar]

Thanks for your reply. Don't want or need to get paid for it. My "payment" is a lively discussion on FR with the hope that people no longer just accept polls at face value. I think those who are arguing against my case study's accuracy because of how simply I took the MOE's miss the point — I'm reflecting the average person's viewpoint, not the scientist or pseudo-scientist.

[RatSlayer]

you said:
"Fourth, my premise, which is completely true, is that there is no objective way for the general public to verify the accuracy of most polls."

On this we agree. But, you have tried to take your results and claim that they are valid for all polls generally. In order to do this, you need more than a single data point (i.e. more than a result of single election). And even then, it would apply only to election polls and not other types of polls.

Why more than one election results? Because there are hundreds of reasons why a poll could be wrong and some of these reasons could cause all the polls for a single election to be inaccurate. But some other election may not suffer from this problem.

Basically, you did the equivalent of using the LA Lakers to study the average height of all humans. Yes your results can be valid for the LA Lakers (the CA recall), but not for the general public (all polls in general) and possible not even be valid for other NBA teams (other election polls) such as the team that used to have Buggsie Malone.

Therefore, you must have polls from more than one election result to determine how often these reasons occur and how much they affect the accuracy of the results.

you said
"Sure it would be nice to have the time and resources you describe, with hundreds of elections and polls to study and computer programs to sift through them, "

The point I was trying to make is that to support your conclusions, it would require a study of such magnitude it would qualify for a Master's Thesis, perhaps even a PHD in statistics.

The ironic part of all this is that the type of study you are attempting is a close cousin to a poll, because they are both attempts to apply statistical analysis to a data set and thus are very dependant on a good data set.

[Wolfstar]

Thanks for your posts, Doc. You may very well be right in that the way MOE's are explained or used by the media and pollsters gives an incorrect impression of their true meaning. But I, absorbing the information as "everyman," take the media and pollsters at face value. If they say the percent a poll came up with for a question is accurate plus or minus several percentage points, that forms the basis I, as "everyman," will use to compare against an actual election result.

Don't you understand? My point is that there is no way whatsoever for "everyman" — the ordinary person running around taking care of their busy lives — can possibly verify the accuracy of polls. When I am told by Gallup, or CNN/Time, or USA Today, or Zogby, or any other organization what the totally artificial "presidential approval rating" is today, how the heck am I supposed to know if it's true? By what objective criteria? Because somebody throws some mumbo-jumbo about an MOE at me? You're talking to "everyman," who frequently can barely balance a checkbook and make ends meet.

Don't you get it? If "everyman" can take a bunch of polls, do some simple arithmetic at the "balance-the- checkbook-level" and find what he understands to be major inaccuracies, then something is very wrong with the whole house of cards that is polling today. It may be, as you point out, not on the statistical side, but in the way polls are analyzed and reported. I as much as said the same thing in my own analysis above: whether what the arithmetic showed was due to methodology or ideology could not be determined.

I don't know of any other "science" that is unverifiable. Do you?

[Wolfstar]

But, you have tried to take your results and claim that they are valid for all polls generally.

Again, this was a CASE study. The case being the California recall and how 20 polls did in predicting the outcome. I am not claiming anything other than that the results of this study demonstrate that polls can't be trusted. How is the general public to know which poll is accurate and which is not. What benchmark can they use? Why should they take your word, or my word, or anyone's word that some polls are accurate? Which ones?

I do agree that this case study is similar to polling in this way: Polls take a small sample of respondents and use them as a case study from which to extrapolate opinions held by the public at large. If small sample size is a flawed approach for a case study, certainly the same must hold true for polling.

[Wolfstar]

Midday ping. Interesting discussion. If this thread just gets folks to thinking about the central point — that there is no objective way to verify the accuracy of most polls — then all this effort will have been worth it.

[PhiKapMom]

Thanks for posting this information! Much appreciated!

[KayEyeDoubleDee]

Let's use an example of 45% +/- 3%. The pollster can claim to be right if the percent of the vote comes out at 42, 43, 44, 45, 46, 47, or 48 percent.

I'm sorry, but this simply is not true. For example, please read this on MOE.

[blackdog]

Polling data is a product just like the tangible ones such as cars, appliances, or clothes. Just what happens if you are in the unfortunate position as a polling sales person and you know your customer who is willing to pay you handsomly for today's business and has done so in the past, and you know the data you are going to supply is devastating or at best very bad news?

You put lipstick on the pig, cash the check, and start writing spin scenareo's for when the phone calls come. Wallstreet analysts operated this way for an entire decade. It's a pollster's turn now. The only problemo is that the Rat nest has a tendancy to shoot the messenger. So look for more of the same cooked up poll results to provide adequate levels of customer satisfaction.

[Wolfstar]

Thank you very much for going to the trouble of finding that information and providing a link. I read it carefully and find that it just reinforces my point. I know my take on it will surprise you, but bear with me.

The linked info used a CNN/USA Today/Gallup poll re Clinton-Dole to highlight how MOE's are incorrectly reported:

"The CNN/USA Today/Gallup poll taken June 27-30 of 818 registered voters showed Clinton would beat his Republican challenger if the election were held now, 54 to 39 percent, with seven percent undecided. The poll had a margin of error of plus or minus four percentage points."

The "everyman" I've been trying to give a voice to hears or reads that and internalizes it as I previously described, that is, plus or minus 4 percentage points for Clinton's 54, and plus or minus 4 for Dole. The report describes both the poll results and the MOE in percentage terms. What other conclusion is anyone other than a mathematician or a statistician going draw?

Niles goes on to write:

"In this case, the CNN et al. poll had a four percent margin of error. That means that if you asked a question from this poll 100 times, 95 of those times the percentage of people giving a particular answer would be within 4 points of the percentage who gave that same answer in this poll."

Unless I am completely missing his point, in a different way he's saying the same thing I did — with the exception that he adds information about asking the same question 100 times and expecting that 95 times the resulting percent would be within four points of the first result. He does not clarify the use of plus or minus, but I don't see any other way to interpret it except that, using his example, it's +4 and -4 percentage points over or under the original percent. If there is another interpretation, please let me know.

In the recall case study, as already mentioned, I've done exactly what many, many people — including polling firms, themselves — do after elections. Compare a pollster's percentages to actual election results. If this is a flawed way to guage the accuracy of the relative handfull of polls that can be checked against actual elections, then a great many people labor under exactly the same misconception.

[Wolfstar]

Polling data is a product just like the tangible ones such as cars, appliances, or clothes.

Most assuredly yes. However, unlike cars, appliances, etc., and even the service professions such as medicine, law, accounting and the like, the consumer has a way to evaluate the quality of the product. Polling is an altogether different kettle of fish, because both the consumer and the reporter of polls haven't got a clue whether or not any given poll is accurate. Yet polls intrude into how our laws are made, how we are governed, who we vote for and how our elections are conducted, and so on. In media hands, they help shape and drive real public opinion. So challenging them is a healthy thing.

[yoe]

Thank you! However we had eight years of a president (s) running this country by POLLS and the media treated them as if they were pound sterling - when often just the opposite was happening and unreported.

One of the best things about FR is that you can always find the truth and facts from our posters - truly a pound sterling bunch with some pretty enviable knowledge.

[Southack]

Nice!

[Wolfstar]

Thank you both. Polls are sort of like chocolate chip cookies — great fun to consume, but not good for you in the long run.

[RatSlayer]

you said:

"I do agree that this case study is similar to polling in this way: Polls take a small sample of respondents and use them as a case study from which to extrapolate opinions held by the public at large. If small sample size is a flawed approach for a case study, certainly the same must hold true for polling."

Yeah, now your starting to get it. I'm not disputing your conclusions. In fact, I think you're right that most polls can't be trusted. I'm just saying that your methodology falls short of what is needed for proof. And that in some ways, it falls short in the same way that polls fall short.

BTW, if you'd like to see some examples of polling that are usually highly accurate, take a look at exit polls. They are generally much more accurate than pre-election polls (or almost any other type of poll for that matter).

[DrDeb]

. . . excellent work -- you've given us all much to ponder!!

[ForGod'sSake]

BTW, if you'd like to see some examples of polling that are usually highly accurate, take a look at exit polls.

Roger Ailes on Exit Polls - Congress...funny.

BTW Woflstar, I think you're take on this is right on. Polls can, and likely do, show generally whatever the pollster thinks they should show. Not unlike our friends in the media, eh?

FGS

[ForGod'sSake]

BTW Woflstar, I think you're take on this is right on.

What I meant to add to my earlier commment was that your findings(?) seem intuitively correct to the great unwashed; that's us. I gathered that's all you were really hoping for.

FGS

[Wolfstar]

Always appreciate thoughtful responses, like yours, that advance the dialogue. Exit polls are more accurate for a good reason — exist pollsters catch real voters just after they've participated in a real election, and ask them how they voted.

[Wolfstar]

Thanks, DrDeb. That's all I can ask — that others think through the issues raised.

[Wolfstar]

What I meant to add to my earlier commment was that your findings(?) seem intuitively correct to the great unwashed; that's us. I gathered that's all you were really hoping for.

Yep. Trying to give a voice to "everyman" whose suspicions about polling go totally ignored by the elites who keep trying to tell him how he "feels."