WHICH POLLSTER WAS THE MOST ACCURATE? (ranks both national and state pollsters)

RealClearPolitics ^ | 11/8/04 | T. Bevan & J. McIntyre

[Cableguy]

By and large the pollsters did an excellent job this year, especially when you consider the significant variables they were grappling with (massive registration increases, historic levels of intensity, cell phones, etc) and also that they were working under intense public scrutiny and scorching partisan attacks from both sides.

At the national level the answer to the question is pretty straightforward. Ed Goeas's GW-Battleground Vote Projection and Pew Research got it exactly right. Goeas's final Battleground projection was Bush 51.2, Kerry 47.8, Nader 0.5 and Pew's final allocation was 51-48-1.

CBS News/NY Times also nailed the final spread in the race (Bush +3), though they didn't allocate undecideds which makes their final less impressive than Battleground's and Pew's.

Raghavan Mayur at TIPP also deserves a mention for outperforming almost all of the big media pollsters and coming in just a tick off the final results with their final Bush 50.1, Kerry 48.0, Nader 1.1 projection. Scott Rasmussen of Rasmussen Reports also had a solid final result with their Bush 50.2, Kerry 48.5 final projection.

USA Today/CNN/Gallup finished poorly this year. Ironically, Gallup would have finished at or near the top of the list had they allocated undecideds 50/50 like Pew, as opposed to giving what appears to be 100% of the undecideds to Kerry.

By the way, just as a point for future reference Bush's Job Approval rating among likely voters in Gallup's final poll was 51%. President Bush's percentage of the national vote total this election? 51.0%.

Of the entire group of pollsters this year, only three got it wrong. Celinda Lake of Lake, Snell, Perry & Associates (the Democratic polling firm for the GW-Battleground 2004 Poll), Marist College and FOX News/Opinion Dynamics all projected a John Kerry popular vote victory. (The Democratic polling firm Democracy Corps also called for a one-point Kerry victory.)

Poll	Bush	Kerry	Nader	Error
Final Results	51.0%	48.0%	0.4%	-
Battleground/Tarrance	51.2	47.8	0.5	Dead On
Pew Research	51	48	1	Dead On
CBS/NY Times	49	46	1	0%
TIPP	50.1	48.0	1.1	1%
CBS News	49	47	1	1%
Rasmussen	50.2	48.5	-	1%
ABC/Wash Post	49	48	0	2%
Harris	49	48	1	2%
NBC/WSJ	48	47	1	2%
Reuters/Zogby	48	47	1	2%
Newsweek	50	44	1	3%
CNN/USAT/Gallup	49	49	1	3%
ARG	48	48	1	3%
Marist College	49	50	0	4%
Battleground/Lake	49	51	0	5%
FOX/Opn Dyn	46	48	1	5%

Now to the battleground state polls. Let's stipulate up front that both the national poll and the battleground state analyses are not meant to be comprehensive or quantitative, but rather quick snapshots using a simple, commonsense grading system. We looked at two basic factors to determine accuracy rankings: the percentage of states where the pollster correctly picked the winner and how much the pollster's projections varied from the final result in each state.

One last thing. In addition to calculating the difference between the projections from each polling firm and the final vote totals, we also tried to determine whether we could detect any trends that might show favorability toward one party or another by consistently underestimating or overestimating support for one candidate or another.

Here is what we found from all of the final polling conducted in the 2004 battleground states at the Presidential level (Click Here to View the Full Tabulated Results):

1) Mason-Dixon
Failed to Project Winner: 6.2% | Average Error = 1.8

A final Minnesota poll showing a one-point Bush win is the only blemish on Mason-Dixon's otherwise perfect scorecard this year. Not only did Brad Coker project the correct winner in 15 out of the 16 battleground states we looked at, he did so with amazing accuracy. Four states were dead on the final number and the overall difference between Mason-Dixon's final polls and the actual election results was a minuscule 1.8 points. Furthermore, if you look down through the list of Mason-Dixon's projections it's impossible to detect any consistent leanings toward one candidate or another.

2) Rasmussen
Failed to Project Winner: 6.2% | Average Error = 2.3

Rasmussen's battleground state polling this year was extremely solid and a close second to Mason-Dixon. He was dead on in PA and just missed in IA. Average error was a very good 2.3 points, with NJ and AZ the only states where he wasn't within three points of the final spread. No partisan trends either way.

3) SurveyUSA
Failed to Project Winner: 7.1% | Average Error = 2.8

Some people have questioned methodology and reliability of SurveyUSA's polls. Their performance in the battleground states this year should answer a good number of those questions: 13 out of 14 states called correctly, including dead-on numbers in Maine, Michigan, and Ohio. The difference between projections and actuals in a few of the states (CO, FL, NV & NJ) was on the high side, even if it was within the margin of error. No identifiable leanings toward either candidate.

4) Research 2000
Failed to Project Winner: 14.3% | Average Error = 3.1

Del Ali's firm only conducted polling in seven battleground states this year. They got six of those states right, including nailing a Bush one-point victory in Iowa. The big miss came in Florida, where Research 2000's final poll called for a one-point Kerry win. Average error from the final results was 3.1 percent - which is respectable - though in every instance except one (Iowa) they overestimated support for Kerry and underestimated support for Bush.

5) Quinnipiac
Failed to Project Winner: 33.3% | Average Error = 2.3

Many people dismissed Quinnipiac's final poll in Florida (Bush +8) as an outlier. Wrong. Quinnipiac was closer than most in FL, and they also did a nice job in the only two other states where they polled, NJ and PA. In fact, Quinnipiac would have finished pretty high in our rankings except they called for a tie in Pennsylvania. All three of their projections overestimated the spread for President Bush.

6) Zogby
Failed to Project Winner: 27.3% | Average Error = 3.8

As we all know, Zogby had been on record for months saying that Kerry was going to win this race. Despite his final tracking poll that put Bush ahead by one point nationally, Zogby's polling at the state level reflected his belief that Kerry was going to be the beneficiary of huge turnout - especially among the youth vote. The result is that Zogby missed three of the eleven states he polled in (FL, IA, and NM), had a relatively high error rate across the board (3.8%), and his numbers generally skewed in favor of John Kerry.

Adding insult to injury, Zogby's bizarre election day antics calling for "surprises" in Colorado and Virginia and a decisive 311 electoral vote victory for Kerry suggest he was relying on (not to mention taken in by) the badly skewed early exit poll data.

Let's be honest: Zogby's conduct this year bordered on outrageous. No other independent pollster was out making public predictions of a John Kerry or George W. Bush victory months before hand. And no other pollster decided to wait until 5:30pm Eastern time on election day to post their final numbers.

7) American Research Group
Failed to Project Winner: 50% | Average Error = 2.0

ARG got a bit unlucky this year. They called for a 1-point Bush win in NH and the result was Kerry +1. They also projected a 1-point Kerry win in New Mexico and the result was Bush +1. The big miss, however, came in Florida where ARG's last poll had Kerry up two. ARG did offset these misses by nailing tight outcomes in IA and WI giving them an excellent score in overall average error.

8) FOX News/Opinion Dynamics
Failed to Project Winner: 50% | Average Error = 4.5

It stands to reason that if your national numbers are way off then at least some of your state numbers are going to be bad as well. This is certainly true of FOX's final poll in FL where they showed Kerry ahead by five points. In fact, that 10-point miss stands out as the worst among the final battleground polls we looked at. Another miss in Wisconsin put FOX News/Opinion Dynamics at 50/50 in battlegrounds, with an average error rate of 4.5%

9) Strategic Vision
Failed to Project Winner: 44% | Average Error = 2.4

We were assaulted by some people for labeling Strategic Vision a Republican polling firm. For the record, the reason we did this was twofold: 1) they had a history of polling for Republican clients and 2) their polls were not sponsored by any independent media outlets like newspapers and television stations.

Strategic Vision's projections for a Bush win in MI and a tie in NJ, seems to provide evidence that, at least in these two states, they were skewing toward President Bush. Their polling in the other seven battleground states was reasonable, though except for Florida and Ohio, they consistently underestimated support for Senator Kerry.

10) CNN/USA Today/Gallup
Failed to Project Winner: 67% | Average Error = 5.8

To find Gallup's name at the bottom of the list is nothing short of shocking. In four of the biggest, most important states in the election this year (FL, OH, PA, and WI) CNN/USA Today/Gallup wasn't even close. In fact, they got it exactly backwards calling for Kerry wins in Florida and Ohio by 3 and 4 points, and Bush wins in PA and WI by 4 and 8.

 

[Cableguy]

ping

[demlosers]

For state projections I'd say Mason-Dixon.

[big'ol_freeper]

Where's Zogby? Oh, he is at the bottom of the list. Seeee ya!

[CWOJackson]

Sorry guy but you left out one of the most accurate polls this election: the 7/11 coffee cup poll.

7-Election Outcome Reflects Current Actual Election Results Dallas, TX, November 3, 2004 - Today, millions of Americans will pour their first cup of the coffee of the day and pick up a newspaper at 7-Eleven stores to read the results of the too-close-to-call presidential election. And during the month of October, these same Americans predicted the outcome of the election with their coffee purchases with 7-Election, the coffee cup poll sponsored by 7-Eleven, Inc., the world’s leading convenience retailer.

Final 2004 7-Election cup counts tracked identically with published national election results of 51 percent for Bush and 48 percent for Kerry, and within a few percentage points of actual poll results in many states. Casting their “votes” in participating 7-Eleven stores across the country during the month-long promotion in October, customers served up these results:

I’M VOTING FOR BUSH cups – 51.08 percent

I’M VOTING FOR KERRY cups – 48.92 percent

Among swing states, Sen. Kerry had the top 7-Election cup count in Ohio, Pennsylvania, Michigan, Colorado and New Hampshire, with President Bush prevailing in Florida and Nevada. Some states’ 7-Election results were less than a single percentage point apart. National, state and metropolitan results of the month-long poll were reported on the company’s website (www.7-Eleven.com) Monday and, just as in the 2000 election, proved to be about as accurate as the official voter polls sponsored by various media outlets and professional research companies. In the 2000 7-Election, President George W. Bush “out-coffee-cupped” Vice President Al Gore by just one percentage point.

“Some predict the future with tea leaves, I prefer coffee cups,” said Jim Keyes, president and CEO of 7-Eleven, Inc. “We have no margin of error because the whole idea of 7-Election was to have some fun and promote voter participation, not to provide a statistically sound sampling. However, it is ironic, that our results do fall within those historical margins of error or plus-or-minus 3 percent.”

With more than one million customers buying coffee in 7-Eleven stores nationwide each day, the company sells more hot beverages than any of its other proprietary products, brewing more than 10,000 pots of coffee an hour. Last fall, 7-Eleven introduced Café Combinations, expanding its hot beverage selection to include flavored syrups, toppings, new cappuccinos and cocoas, and steamed milk mix – enough options to create more than 1,300 different hot beverage combinations.

___________________________________________

Screw Zogby. When I want to know what is happening I'll ask Habib at 7/11.

[bad company]

I believe PEW was the closest in 2000 too.

[george wythe]

Great analysis.

I think that real winner was the average of all the polls, since the margin of error shrinks significantly when several polls using disparate methods are averaged.

[trashcanbred]

FOX News projected a Kerry popular vote win? What a bunch of liberals! </sarcasm>

[Tribune7]

bump

[Lou L]

Too bad, no mention of the accuracy of the exit polls.

[Blue Screen of Death]

My uncle Fred was right on. He drank a beer and checked a woolly caterpillar. Then he drank another and watched the pattern of ducks flying north. He had another beer and looked at the lichen on the north side of the well. Finished the six pack and burped. The burp sounded more like Bushhhh than Kerryyyy so he declaired George the winner and fell asleep on the porch.

[My2Cents]

So much for Gallup being the "Gold Standard." They did worse than Zogby.

[MattinNJ]

FReeper Dales was actually the best. I printed out his final projections from his website and used it as a scorecard all election night. As state by state results came in, I just kept checking them off his scorecard.

I think the only state he missed was Wisconsin, which might still be counting the absentee ballots.

[Cableguy]

Actually, Opinion Dynamics are run by bunch of former Democrats.

[Cableguy]

No, the new "gold standard" is now Pew and Battleground, plus Mason-Dixon, Rasmussen and SurveyUSA for state polls.

[Born Conservative]

bttt

[marblehead17]

ping

[BlackRazor]

Personally, I don't think a polling firm should be judged by whether they accurately "project" a winner, if the actual results are within the margin of error of the poll. For example, ARG picked Bush to win NH by 1 point. Kerry actually won by 1 point. ARG was pretty darned close and shouldn't be penalized in my view. They got it right within the margin of error. IMO, pollsters should only be evaluated based on average candidate error, not by whether they get winners right.

[My2Cents]

We all gave Rass a pretty rough time this year. I'm glad he was able to redeem himself. (But I still think publishing poll results for months on end that were never outside the margin-of-error was a waste of bandwidth.)

[My2Cents]

The big reason Pew was right-on? Luck!