Are political polls accurate? CA recall study of 20 polls says...NEVER TRUST POLLS AGAIN!

RealClearPolitics.com & original material ^ | 10/21/03 | Wolfstar

[Wolfstar]

Ah, the omnipresent poll. The media punditocracy is addicted to using polls to tell us what "the American people feel" (never think) about everything from a president's so-called approval rating, to how a candidate's chances stack up against others in a race, to our "feelings" about various policy and social issues. Although the public has absolutely no way to evaluate the vast majority of polls for accuracy, most of us simply accept them as incontrovertible indicators of truth. Why? The answer probably is because we're told that polls are "scientific" since they use statistical-type analysis, and most of us tend to equate science with a search for objective truth.

Despite the scientific patina of a Margin of Error, how does one incontrovertibly prove a poll to be accurate? For nearly all polls, the answer is that it can't be done. In the political realm, only polls taken closest to an election can be compared for accuracy to the actual results of real voters expressing their real attitudes at the ballot box.

Nevertheless, the public is conditioned not to look too closely at the accuracy of polls. We're told polls are "just a snapshot in time." So we learn to shrug when there are big swings in, say, a president's "approval ratings" from one month to the next even though nothing particularly new happened during that time. Despite the fact that only a handful of polls close to elections can be verified against objective results of real votes, polls affect our society in several ways. The worst is that many politicians, hanging on their every nuance, too often make crucial policy decisions based on polls wielded like clubs by the media and special-interest groups.

So what's the truth about polls? Few in the public have the time or resources to do an in-depth study in an attempt to answer that question, while those who butter their bread on the back of polls have no incentive to do so. Ah, but fate stepped in this year in the form of the California Recall and provided an excellent opportunity to do a case study of multiple polls taken in a highly compressed period of time. Since all poll results in this study were obtained relatively close to Election Day, a comparison to actual election results is not only easier, but also more instructive than, say, an apples-to-oranges comparison of a year's worth of "approval ratings" to an election outcome.

This study encompasses 20 polls taken by nine polling organizations between Aug. 7 and Oct. 5, 2003. It looks at the five most-watched poll questions — those which purported to measure the percent of vote for:

• Yes on recall
• No on recall
• Arnold Schwarzenegger
• Cruz Bustamante
• Tom McClintock

Because there are 20 polls and five questions, this study encompasses 100 individual results. When analyzed by several criteria, such as poll date and accuracy for each of the five questions, an eye-popping picture of polling precision — or lack thereof — comes into sharp focus. Note that "accurate" in this study means: (1) a poll result within that poll's MOE, and (2) as compared to the actual election results. "Inaccurate" or "wrong" means a poll result outside that poll's MOE as compared to the actuals.

Q: Are political polls accurate?
A: Based on this study, the answer is: While some results for some questions in a multi-query poll may be accurate, most polls, when taken in their entirety, are not. Here's why: Of the 100 individual question results, more than half (57) were wrong (outside their MOE's), as compared to the actual election results.

However, the stunning fact is that only 1 poll in 20 gave accurate results across-the-board for all five questions. This was the last poll taken by Survey USA from Oct. 3-5. Only 4 of 20 got both the Yes and No on recall questions right, while only that last Survey USA poll gave accurate Candidate results. In other words:

• Yes/No: 16 of 20 polls (80%) got either the Yes question or the No question right, but not both.

• Candidates: Only one poll (5%) gave accurate results for all three candidates. Most importantly, many of the inaccurate results were wildly wrong and served only to totally skew the picture fed to the public by reporters on the presumed status of the race.

Looking at accuracy another way, of the 100 individual question results, less than half (43) were right within their poll MOE's. Most (67) under-estimated the actual election results, and only seven poll questions called that election result precisely.

Q: Does averaging several polls over a period of time give a more accurate picture?
A: Some pollsters, reporters and others who rely on them believe either a tracking poll, or an average of several polls taken over a period of time, are the best methods of obtaining an accurate picture of public attitudes. Due to the compressed two-month recall campaign, all 20 polls, taken together, constitute a form of tracking poll. Yet, as already noted, their often wildly inaccurate results only contributed to a false week-to-week perception of the race. So would averaging the results of all 20 polls give a more accurate picture? The answer based on this study is a qualified yes. Here's why. First, all 20 MOE's were averaged to establish a baseline, which works out to ±3.6%.

• For Yes, averaging the 20 polls gives 57%, which is 2 points over the actual and within the average MOE. So far, so good. But, for No, averaging all results gives 39%, which is 6 points under the actual and outside the average MOE. That's 50-50 — or as good as a coin toss.

• Candidates: Averaging 20 polls gives 34% for Arnold Schwarzenegger — a whopping 15 points under his actual percent of the vote and obviously well outside any MOE. On the other hand, the average for Cruz Bustamante is 28%, four points under his actual and just outside the average MOE. For Tom McClintock, the average is 13%, precisely the percent of the actual votes he received. Averaging just misses getting two of three right here.

Q: Is the Margin of Error (MOE) really useful in assessing a poll's accuracy?
A: Based on this study, the answer is a resounding NO! The smallest MOE given was ±2%; the largest ±5.6%. Interestingly, ±2% was for one of the earliest, most inaccurate polls, while ±5.6% was for the last and most accurate.

• Yes: Of the 20 polls, 11 (55%) were accurate (within their stated MOE's) when compared to the actual election result.

• No: Only 6 of 20 (30%) were accurate.

• Arnold Schwarzenegger: Repeat, ONE result was accurate. Most vastly underestimated his vote strength. Pollsters routinely gave an MOE of ±3 or 4 points, yet just as routinely came in with double-digit under-estimates. The worst was the Public Policy Institute poll of Aug. 8-17, which gave an MOE of ±2%, but underestimated the actual result by 26 points! If you're tempted to say, "Well, that's because it was an early poll," nope, not so. As late as Sept. 25-27 (Gallup) and Sept. 25-29 (L. A. Times), two polls with MOE's of ±3% both underestimated the actual vote for Arnold by 9 points.

• Cruz Bustamante: Only 6 of 20 (30%) were accurate.

• Tom McClintock: 13 of 20 (65%) were accurate. (Note that the results for Tom McClintock were the most accurate across all dates and pollsters. Even when wrong, they were wrong only by a small margin. See poll bias question for further discussion.)

Q: Do polls become more accurate closer to an election?
A: The broad answer is a qualified yes — qualified because, in this study, the polls were inconsistent on this question. Results for some questions in early polls were quite accurate, while some late poll results were very inaccurate. However, the trend was to become more accurate closer to the election.

As already noted, the last poll was the only one that got all five questions right within its MOE. The following table shows the total poll questions that the first/last seven polls got right within each poll's MOE. An accuracy of 100% in this instance would be 35 questions right (7 polls x 5 questions). Note that just under twice as many question results were right in late polls than in early ones. Nevertheless, even the late polls (last two weeks) got less than half (49%) of the questions right.

First 7 polls (8/8 to 9/8)		x	Last 7 polls (9/24 to 10/5)
Question	# Poll Questions Within MOE	x	Question	# Poll Questions Within MOE
YES	2	x	YES	5
NO	1	x	NO	2
AS	0	x	AS	1
CB	1	x	CB	4
TC	5	x	TC	5
TOTAL	9 (26%)	x	TOTAL	17 (49%)

Q: Are political polls biased?
A: If any given poll is biased, the hard question to answer is whether or not it is due to ideology or methodology. Every expert on polling says that variables such as the way a question is worded; who the respondents are; the order of questions; even what time of day/week a poll is taken can create a bias. (Many polling organizations do not make their methodology public.) As the following demonstrates, an argument can be made either way for these California recall polls:

• Yes/No: The polls tended to over-estimate the strength of the Yes vote, with 12 of 20 results (60%) over the actual election outcome. The opposite is the case for No, with a whopping 18 of 20 (90%) giving results that underestimated actual. If this odd disparity was due to an ideological bias, why would polling firms want to consistently under-estimate the strength of the No on recall vote?

• Candidates: Here's where a comparison of poll results to the actual election is especially shocking. When one looks at the results for Arnold Schwarzenegger vs. Tom McClintock, the contrast is amazing. It can't be emphasized enough that ALL the polls underestimated the strength of the vote for Arnold - most by a statistically vast margin. The most accurate was 3 points under. The least accurate was 27 points under the 49% of the vote he actually received. An astounding 13 of 20 polls (65%) underestimated his actual vote by a double-digit margin.
  
  In contrast, the reverse happened for Tom McClintock. Half the polls over-estimated his vote strength and three called it precisely - that is, 65% either called McClintock's vote exactly or overestimated it. As can be seen in the Candidate table below, in August the polls tended to slightly underestimate McClintock's vote compared to his actual. In mid-September, they began going the other way and overestimated it by from 1 to 5 points right up to the last Survey USA poll, which called his 13% exactly.
  
  Interestingly, although far, far less severely than those for Schwarzenegger, the polls also tended to underestimate Bustamante's strength with the voters, with 15 of the 20 (75%) giving results under his actual election outcome.

So are political polls biased? Whatever the answer, the staggeringly inaccurate polling for Schwarzenegger — and moderately inaccurate results for Bustamante — as compared to the surprisingly accurate, even slightly inflated results for McClintock certainly should raise a lot of eyebrows. Of the five poll questions in this study, results for Arnold Schwarzenegger were by far the most inaccurate, while those for Tom McClintock were the most accurate. Was there really a mid-to-late September surge for McClintock? Or, as many suspected, were the polling organizations trying to inflate impressions of his strength as compared to that for Schwarzenegger? And did the polls underestimate Bustamante's vote strength in order to boost Gray Davis? One would be tempted to say "yes" to these questions were it not for the fact that 90% of the polls also underestimated the No on recall vote.

Q: Which polling organization was the most accurate?
A: The following table speaks for itself, although the reader is encouraged to take particular note of the poor performance of two big national polls, Time/CNN and Gallup.

Polling Organization	# of Polls Taken	Total # of Results (# Polls x 5 Ques.)	# of Correct Results	Percent Correct
Knight Ridder	1	5	3	60%
Los Angeles Times	3	15	9	60%
Survey USA	5	25	13	52%
CA Chbr of Commerce	2	10	5	50%
Field Poll	3	15	7	47%
Time/CNN	1	5	2	40%
Public Policy Institute	2	10	3	30%
Gallup	2	10	1	10%
Stanford U.	1	5	0	0%

Q: What conclusions can be drawn from this study?
A: Even the most accurate polls in this study were wrong 40% of the time overall (based on above accuracy table). The accuracy of each of their internals was worse. So, when the national media tout polls from Gallup, Time/CNN, Newsweek, Zogby, and such about what "the American people feel" regarding something insubstantial like "presidential approval;" or whether or not they want to re-elect the president; or which issues are most important to them; or how a person who's name is all but unknown nationally suddenly becomes "the frontrunner" for a party's nomination, it's wise to keep three things in mind:

1. There is no objective way to verify the accuracy of most polls.
   
   
2. It is part of human nature to want to predict (thus control) the future. However, this study demonstrates unequivocally that, whether or not it's due to political bias or flawed methodology, polls often drastically misinform the public.
   
   
3. Only 1 in 20 polls in this study got all five questions right. In other words, 95% polls were wrong on one or more of their questions. So when a pollster uses the technique of summing one individual internal question result to another in order to claim something about public opinion, all the pollster may be doing in reality is compounding errors. For example, when Zogby adds answers for, say, "fair" and "poor" together, if either the result for "fair," or the one for "poor," or both are wrong, all he is doing is compounding errors and giving false information to the media and public.

Notes for Tables of Results:

1. The 20 polls and their MOE's were obtained through RealPolitics.com, and Google searches for those where the RealPolitics.com links no longer worked.
2. Results are as of Oct. 20 with 100% of precincts reporting.
3. Over/Under = number of points over (+n) or under (-n) the actual election result.

YES/NO Table Of Results:

Final Results		55%			45%
Poll	Date	YES	+Over -Under	Within MOE	NO	+Over -Under	Within MOE	MOE
Survey USA	Oct. 3–5	57%	+2	Y	43%	-2	Y	±5.6%
Knight Ridder	Oct. 1–4	54%	-1	Y	41%	-4	N	±3%
Field Poll	Sep. 29–Oct. 1	57%	+2	Y	39%	-6	N	±4.8%
Survey USA	Sep. 28–30	61%	+6	N	39%	-6	N	±3.7%
Los Angeles Times	Sep. 25–29	56%	+1	Y	42%	-3	Y	±3%
Gallup	Sep. 25–27	63%	+8	N	35%	-10	N	±3%
CA Chbr of Commerce	Sep. 24–25	53%	-2	Y	41%	-4	N	±3.5%
Survey USA	Sep. 19–22	57%	+2	Y	42%	-3	Y	±3.5%
Public Policy Institute	Sep. 8–17	53%	-2	Y	42%	-3	Y	±3%
Los Angeles Times	Sep. 6–10	50%	-5	N	47%	+2	Y	±3%
Survey USA	Sep. 6–8	62%	+7	N	37%	-8	N	±3.7%
Field Poll	Sep. 4–7	55%	exact	Y	40%	-5	N	±4.5%
CA Chbr of Commerce	Sep. 1–4	52%	-3	Y	41%	-4	N	±3.1%
Stanford U.	Aug. 29–Sep.8	62%	+7	N	38%	-7	N	±3.4%
Survey USA	Aug. 23–25	64%	+9	N	35%	-10	N	±3.7%
Los Angeles Times	Aug. 16–21	50%	-5	N	45%	exact	Y	±3%
Field Poll	Aug. 10–13	58%	+3	Y	37%	-8	N	±5%
Public Policy Institute	Aug. 8–17	58%	+3	N	36%	-9	N	±2%
Time/CNN	Aug. 8	54%	-1	Y	35%	-10	N	±4.3%
Gallup	Aug. 7–10	69%	+14	N	26%	-19	N	±4%

Average of 20 polls		57%	+2	Y	39%	-6	N	3.6%
# Results within MOE				11			6
# Results outside MOE				9			14
# Same as actual			1			1
# Over actual			12			1
# Under actual			7			18

CANDIDATE Table Of Results:

Final Results		49%			32%			13%
Poll	Date	Arnold S.	AS +Over -Under	Within MOE	Bustamante	CB +Over -Under	Within MOE	McClintock	TM +Over -Under	Within MOE	MOE
Survey USA	Oct. 3–5	46%	-3	Y	34%	+2	Y	13%	exact	Y	±5.6%
Knight Ridder	Oct. 1–4	37%	-12	N	29%	-3	Y	15%	+2	Y	±3%
Field Poll	Sep. 29–Oct. 1	36%	-13	N	26%	-6	N	16%	+3	Y	±4.8%
Survey USA	Sep. 28–30	45%	-4	N	28%	-4	N	16%	+3	Y	±3.7%
Los Angeles Times	Sep. 25–29	40%	-9	N	32%	exact	Y	15%	+2	Y	±3%
Gallup	Sep. 25–27	40%	-9	N	25%	-7	N	18%	+5	N	±3%
CA Chbr of Commerce	Sep. 24–25	35%	-14	N	31%	-1	Y	17%	+4	N	±3.5%
Survey USA	Sep. 19–22	39%	-10	N	32%	exact	Y	18%	+5	N	±3.5%
Public Policy Institute	Sep. 8–17	26%	-23	N	28%	-4	N	14%	+1	Y	±3%
Los Angeles Times	Sep. 6–10	25%	-24	N	30%	-2	Y	18%	+5	N	±3%
Survey USA	Sep. 6–8	39%	-10	N	29%	-3	Y	16%	+3	Y	±3.7%
Field Poll	Sep. 4–7	25%	-24	N	30%	-2	Y	13%	exact	Y	±4.5%
CA Chbr of Commerce	Sep. 1–4	28%	-21	N	33%	+1	Y	12%	-1	Y	±3.1%
Stanford U.	Aug. 29–Sep. 8	40%	-9	N	28%	-4	N	8%	-5	N	±3.4%
Survey USA	Aug. 23–25	45%	-4	N	29%	-3	Y	11%	-2	Y	±3.7%
Los Angeles Times	Aug. 16–21	22%	-27	N	35%	+3	Y	12%	-1	Y	±3%
Field Poll	Aug. 10–13	22%	-27	N	25%	-7	N	9%	-4	Y	±5%
Public Policy Institute	Aug. 8–17	23%	-26	N	18%	-14	N	5%	-8	N	±2%
Time/CNN	Aug. 8	25%	-24	N	15%	-17	N	9%	-4	Y	±4.3%
Gallup	Aug. 7–10	42%	-7	N	22%	-10	N	13%	exact	Y	±4%

Average of 20 polls		34%	-15	N	28%	-4	N	13%	exact	Y	3.6%
# Results within MOE				1			11			14
# Results outside MOE				19			9			6
# Same as actual			0			2			3
# Over actual			0			3			10
# Under actual			20			15			7

[Wolfstar]

This study is original material created for the information and use of FreeRepublic.com readers and active participants. Please feel free to share the information with others.

[Wolfstar]

Pinging as promised.

[Wolfstar]

Ping, as discussed.

[ambrose]

ping

[Wolfstar]

Pinging you because of your strong interest in the recall.

[Timesink]

Media Shenanigans / Schadenfreude ping

[Timesink]

One question: People's opinions on things do change over time. Am I misinterpreting that you are, to some extent, arguing that polls taken two months before the election are somehow less valid because they did not end up matching with the final election results?

[KayEyeDoubleDee]

What is/are your background/credentials, in terms of statistcs, etc?

[goldstategop]

Nope. Voter surveys conducted AFTER the election showed most voters made up their minds a month ago. How come NONE of the polls caught this? This is another area where they didn't see voter behavior coming.

[Timesink]

Nope. Voter surveys conducted AFTER the election showed most voters made up their minds a month ago. How come NONE of the polls caught this?

Not to defend the pollsters, but if they didn't get the pre-election surveys right, why would the get the post-election surveys right?

(Actually, I guess that's not exactly a defense of the pollsters' abilities at all, is it? Hee hee.)

I'm also a total believer in the average person's inability to remember how they felt on an issue one week in the past compared to today, much less a month ago or more. Lots of people rationalize outcomes in their own minds after the fact.

[Wolfstar]

The study focuses on the accuracy of polls, not on their validity. My underlying premise is that the accuracy of most polls can't be proven. The value of this study is that these 20 polls were taken during a compressed period of time much closer to the election they were predicting than, say, polls that compare President Bush to a generic, unnamed Democrat more than a year out from the 2004 election. It's easier to evaluate these 20 polls for accuracy against the election as compared to presidential "approval ratings," for example. How can anyone prove their accuracy? Against what benchmark? Can't be done.

For a more in-depth answer to your query, look at the question: Do polls become more accurate closer to an election? The answer is that, in this study, they trended toward more accuracy closer to the election, but not strongly. Even at the end, only one poll got all five questions right within it's MOE.

[Wolfstar]

None. I am a consumer of news and information who wants some means to evaluate ephemeral things like polls. However, one does not need a background in statistics to do the simple math required for this study. Either a result was within its poll's MOE or it was not as compared to the actual election results. Not rocket science. In fact, not science at all, but plain old-fashioned athritmatic and observation.

[Wolfstar]

Ooops, that should be "arithmetic" not the misspelled "arithmatic." I've been up waaay too late trying to get this posted. Time to hit the sack.

[onyx]

What a magnificent post. Bookmarked with appreciation and gratitude. Thank you, Wolfstar.

[onyx]

POSSE ping TO FABULOUS AND INFORMATIVE POST.

[KayEyeDoubleDee]

Either a result was within its poll's MOE or it was not as compared to the actual election results

I'm not trying to be a critic here, and I totally appreciate your efforts. However, I am almost positive that, for example, a MOE of 5% means something like "there is a 90% chance that the actual results will lie within +/-5% of the poll results." Anyway, something to think about.

[heleny]

Wow, the LAT got 60%? I know 20% (the Schwarzenegger vote predictions) was always wrong, and many FReepers recognized that, but I didn't know their results on the other four questions were so (relatively) accurate.

If someone analyzed the actual error (instead of categorizing right/wrong), the LAT probably would not be in the top two, since their Schwarzenegger predictions were very far from the actual results.

Thanks for summarizing the results and sharing with us all!

[heleny]

a MOE of 5% means something like "there is a 90% chance that the actual results will lie within +/-5% of the poll results."

Some of the polls stated something like that, but the newspapers or TV news charts often simply say the MOE was 5%, and it's often left out entirely in oral TV/radio news.

Maybe they couldn't predict the dynamics of the actual voters this time, because voters were motivated in ways not typical of the usual elections. If they had realized this, the polls should have claimed an 80% or lower accuracy.

[heleny]

Some polls allowed for an "undecided" category, while some may have pushed the respondents to choose yes/no or to tell their preference in the candidate question. I don't know what effect on the accuracy removing the undecideds might have.

[RatSlayer]

you said
"Lots of people rationalize outcomes in their own minds after the fact."

Truer words were never spoken.