Remember in future elections: SURVEY USA MOST ACCURATE, MASON DIXON VERY GOOD.

Various ^ | 12/12/04 | Me!

[dangus]

I've analyzed the pre-election surveys of the 10 major multi-state polling firms. They were: American Research Group (ARG), Fox, Gallup, Quinnipiac (Q), Research 2000 (R2000), Rasmussen (Ras), Strategic Vision (SV) Survey USA (SUSA), and Zogby.

I found Survey USA to be the best. Of the 35 states which I researched, SUSA polled in 29 of them. They were correct every time. What was truly astounding was their track record in states where the contest was not close. SUSA was largely on their own polling in these states, and such contests are often very unpredictable since creating turnout models is virtually impossible. Nonetheless, SUSA was perfect in six states (Calif., Maine, Mich., NY, Ohio and Ore.), and within two points in 11 more (Col., Ill., Ind., Kent., Iowa, Mary., Mo., Okla., Penna., SC., and Texas) SUSA's worst state was Nevada, where Bush won by only two points whereas SUSA had predicted an eight-point blowout.

Mason Dixon was probably best in the presidential races, edging out SUSA. They were correct in all 22 predictions. Their worst state was also Nevada, and they came significantly closer, missing by four points. They were within the margin of error in every state presidential race. Unfortunately, Mason-Dixon underestimated Republicans in several Senate races, calling South Dakota wrong, and being off by six points in South Carolina. However, to be fair, Mason-Dixon did their last Senate polls much earlier, and Republican candidates uniformally used a strategy of saving up campaign cash for a last minute push.

Gallup, once the gold standard of polls, was horrible. Their national polls showed wild swings of 20 points between surveys, before hitting the final number very close, but they were dreadful on the state level. They were wrong in five of the seven races they polled, predicting an easy win for Bush in Wisconsin, and a solid win in Penna., while predicting big wins for Kerry in Ohio and Florida, and a win for Betty Castor in Florida.

Fox was also quite poor, showing Kerry with a 5-point romp in Fla., where Bush won surprisingly easily. They also showed Castor romping by six over Martinez. They only conducted three other polls.

Zogby showed a persistent leftward bias. They overestimated Kerry in 11 out of 13 states. The closest hit Zogby had was Penna., where they were only 2 points off. They were wrong in three states, but inexplicably they showed Bush having a comparably easy time in Ohio, winning by six.

American Research Group was also presistently left-leaning, getting 4 out of 7 wrong. That might look worse than Zogby, but at least they were close, with Florida being the only state they were as much as 3 points off on. But even though their misses were within the margin of error, it is hard to strike their record up to bad luck... they were persistently left-leaning the entire campaign.

Research 2000 went 8 for 8, but passed on the most closely watched races. Their predictions in the uncontested races weren't terribly good: 6 off in NC, 5 off in Ind., 4 off in Ill., 4 off in Mo. They were, however, perfect in Iowa and Nevada, which tripped up other polling firms.

Rasmussen partly atoned for dreadful showings in 2000 (when they called the popular vote for Bush by nine points) and 2002 (when they skewed persistently left-wing.) Ras was 12 for 12, although their margins were often well off.

Quinnipiac also never called a state wrong, although they called Penna. too close to call. They hit all eight state presidential races, plus they were perfect in S.D., and correct in Col.

Strategic Vision got 2 out of 11 wrong, plus they inexplicably called N.J. too close to call, as well as Minn., so they really only got seven right.

One last note: It’s possible that the media’s premature calling of the race for Kerry did hurt Bush. He underperformed poll expectations in the three pacific time zone states for which multiple last-week polls were taken, Oregon, Nevada and Washington. Out of 11 polls, Bush under-performed in 8 and over-performed in none of them. In most of the rest of the country, he did slightly better than he was expected to. The western states have histories of turning against the candidate they perceive as losing.

[InTheRight]

I seriously doubt that even the liberal media will give much weight to his polls in the next election. His tracking polls this year looked like he was pulling the numbers out of a hat. He had huge day-to-day swings that made absolutely no sense.

[dangus]

You mean the "Battleground Poll," which is a collaboration of Lake/Goeas, the Democratic company, and Tarrance, the Republicans. They only publish national polls, and they seem in the mainstream as far as accuracy. The movement in their polls also seems quite credible. And they are great on methodology.

[Do not dub me shapka broham]

I know.

The New York Post relies upon their polls-among others-for their usual political forecasts.

[dangus]

Yes, actually, I used the Nov.1 numbers from RealClearPolitics for Zogby; I thought I remembered his last-day numbers being even worse.

[InTheRight]

At least Gallup was able to identify a shift in party identification toward the Republicans after the GOP convention. Zogby insisted on weighting his polls to a 3-point or so demo advantage, since that was how voters in 2000 identified themselves. The problem was that on voting day, there were actually 5% (if my memory serves me) more Republicans voting than democrats. Gallup does not weight for party ID, which is what enabled them to notice this sea change.

[Do not dub me shapka broham]

The actual percentages split pretty evenly.

However, those numbers are misleading.

There's a sizable percentage of registered or self-described "Democrats", who will never pull the (D) lever for any national office, especially that of president of the United States.

[InTheRight]

I'm thinking his final prediction was 311 electoral votes for Kerry.

[dangus]

>>Zogby will continue to be hailed by the leftist media as reliable.<<

Dass raht! Zogby be da only one ta tell da chroof dat Kerry beat Bush!

[dangus]

Thanks.

>>but one thing that's always bugged me is that the only way to judge a polling organization's accuracy is to compare its final survey to the actual results.<<

Your're right. First, I crunched the numbers based strictly on the final poll. Only later, did I look at what happened earlier in the races, adding that in after the fact.

The truth is ARG and Zogby were abominable all campaign season, often showing wild Kerry leads. But I had no way to objectively compare such things. For instance, I made myself forget that Zogby at one point predicted Kerry was running away with Tennessee; Bush beat him by ten points. I also ignored his last-day prediction that Virginia was too close to call, because it wasn't in his data.

[dangus]

Oops.. Zogby had Kerry by 10 in TN in June, but the final result was Bush by 15.

[InTheRight]

Okay, I found where I jotted down Zogby's final predictions that he posted on Nov. 2. He predicted Kerry would get 311 electoral votes and Bush would get 213. Nevada and Colorado were too close to call. He predicted the popular vote would be Bush 49.4% to Kerry 49.1%, with Kerry taking Florida, Iowa, New Mexico, and Ohio.

[Texas_Jarhead]

Nice job. Thanks for posting.

[gop_gene]

If I remember correctly, Zogby had Arkansas tied!!! Sure enough, as many knew it would come out, it ended up anything but tied.

[Texas_Jarhead]

"Opinion Dynamics are; their polls are sold exclusively to Fox, and Fox creates the formula, so I don't think it's wrong to call it a "Fox" poll..."

Hmmm, that's very interesting. Are such arrangements common?

[patj]

This is very interesting and thanks for the hard work. Why do you think everyone was so wrong in Nev.?

[geedee]

You done good, Dangus.

One of these days the politico wackos are gonna smell the roses and realize us pajama-clad FReepers are the ONLY analysts worth a damn. Hell's fire, Buckhead already showed cBS they'd best not try to blow any smoke up our patoots any longer. And hundreds of other FReepers, maybe thousands, have exposed lie after lie just during just this past year alone.

And they call us dumb? I'd love for there to be an honest BRAINPAN comparison between their best and brightest versus our own John Huang, MiaT, anniegetyourgun, Grampa Dave, Congressman Billy Bob, you, etc. I don't ALWAYS agree with my fellow FReepers but most of them suckers are MENTAL GIANTS compared to the Liberal mental-midgets like Maureen Dowd and Dan Blather.

Hell, I'm even smarter than I thought I was. LOL. I went here a couple of months ago and took their free, online IQ test and it's quite obvious to me that Free Republic has made me smarter. I made sure to email my results to ALL my family and challenged them to either measure up or join Free Republic.

We're a smart bunch here.

[GoMonster]

Appreciate your analysis. However, may want to spend more time. Survey USA which has had good track record in state by state in the past was at best just ok this year. Really state by state Rasmussen's change model worked with Mason Dixon as the best. SUSA was off in a couple of states. Also, note Fox uses Opinion Dynamics which is ironically run (polling) by a Demorat. Zogby really sucked. Gallup also was pretty bad at the end. Battleground did well also.

[dangus]

>> SUSA was off in a couple of states. <<

No, you're wrong. I have them as perfect. Only way they flubbed a poll is if they conducted a poll in a state no-one else did, but that would mean flubbing a poll that wasn't even contested. Their worst states were Nevada by 6 and New Jersey by 5.

I do agree that Mason-Dixon was the best at the state presidential races, and stated so. I didn't look at Battleground because they only did the natinwide poll, so I only had one data point to evaluate.

Let me say this about Rasmussen: I evaluated the polls based purely on election-day performance. Rasmussen ran a strong 3rd, being off 10 in Alabama, 6 in Arizona and 5 in New Jersey. They took a few early polls (as in Mid-October) which made no sense: they were 10 off in Maryland, but I didn't count it.

But I think Rasmussen gets off easy. I ignored previous years' flops when ranking them third; they don't acknowledge any change of technique. As far as I'm concerned, they are, along with Zogby and ARG, thoroughly non-reputable. The only difference between this year and their previous disasters appears to be that their model (unscientific, and unverifiable) just HAPPENED to be right.

[dangus]

I mentionned it in the article: I think that the Pacific Time Zone was hurt by everyone all-but declaring Kerry the winner while pacific-coast people were still at work. Washington and Oregon also skewed to the left. (although SUSA and Ras were both dead on in CA.)

[dangus]

>>Hmmm, that's very interesting. Are such arrangements common?<<

Yes. The networks tend to have such relationships with their poll sources. Only in the case of the other networks, they did not release state-by-state results like Fox did.