Skip to comments.Poll Math (Fun with Zogby!)
Posted on 10/21/2008 11:58:01 AM PDT by PhilosopherStones
Thought I'd have a little fun with poll math. Zogby was kind enough to provide some data which I have compiled into two tables below.
The original Zogby data (which they use to promote their business, so they must be proud of it) can be found here.
Explanation of Table 1:
Table one shows how accurate Zogby was when compared to actual election results (I've omitted races in which their was a significant third party vote or where Zogby had no polling data posted. Hey, you don't like it? Make yer own DANG spreadsheet!)
Col 1 is just the line number. Col 2 is the race. Col 3 is Zogby's pre-election number for Republicans. Col 4, ditto for Dems. Col 5 is the (absolute) spread between the two candidates according to Zogby. Col 6, 7 and 8 are the ACTUAL election results. Col 9 (E) is the difference between the Zogby results and the election results, ordered by greatest to least. Finally, Col 10 (C) shows whether Zogby called the race correctly (R) or incorrectly (W).
You'll notice that we have to get all the way down to line 39 before we see that the actual results and the Zogby predictions are within 4 points of each other. In other words, more than 60 percent of the time, Zogby's predictions were outside of their own given margins of error.
The other thing to note is that even when Zogby was within the margin of error, they called the race wrong 18% of the time (4 out of 22).
Table 2 is the same as table 1 except ordered by Zogby spread:
Here, look at the races that Zogby found close. Particularly, look at where Zogby had the race closer than 4 points. You'll find that they called the wrong winner a whopping 39% of the time (7/18 races).
Don't let the pollsters get you down. This thing is winnable!
Otherwise, as always, feel free to revise and extend my remarks.
Great work! Thanks for taking the time to sort out the data. This is very helpful for the poll junkies.
election poles ping
election poles ping
Just scanning the numbers, it seems the overriding trend with Z is that he gives more of an an advantage to the Republicans than the actual vote reflected—which is not encouraging.
Too late now. Next time I’ll FReepmail you for suggestions before I post any formatted text.
And Zogby under-estimates Republicans by an average of 4 pts. Surprise! Surprise!
Anyone know just how accurate this site is? It seems way out-of-wack to me.
I’ve been hoping to come across a time series of 2008 Presidential poll data that includes error bands - let’s just be generous and say 2 sigma, although I’d really like to see 3 sigma. My gut feel is there would be no significant trend.
Looks like just the opposite. So it’s good news.
Ummm, by my count (maybe I’m not reading the chart right), the races where Z gave the Republican MORE percentage in his projection than the actual vote were 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 15, 16, 17, 20, 22, 23, 24, 25, 27, 30, 34...need I go on?
I even see Rasmussen himself in interviews misstating this simple fact. I've seen him say things like "Obama is up 5 which is outside our margin of error of 3". Nope, he would have to be up by 7 to be outside the margin of error of 3, because MOE does not measure differences between the numbers, but rather applies to EACH of the candidates's numbers.
So, when you point out that you have to go down to column 39 to see actual results within the MOE of 4, what you should really be saying is that you have go down to column 21 before you find results that exceed the combined candidate MOE of 8.
Still, your point is well taken, 21 out of 61 races were outside the doubled MOE. In theory only 5% should have been outside that figure not 33%.
This can easily be explained though, by the second of the great polling fallacies. MOE only measures "statistical" error. That is error due solely to random sampling chance. There are many other kinds of errors that occur in polls and they are additive to MOE. Those range from 1 to 6 percent *in addition to MOE*, typically 2-4% when studied on major pollsters, far more on less reliable pollsters.
Applying these other sources of error, we find that in this set of polls Zogby had really a fully loaded error of 9.5 points per candidate, or 19 points overall. Only with that figure do we get to the 95% confidence level (95% of the races listed fell within that level of error).
In this set of polls, then, a candidate had to be up by 20 points to be "outside the margin of all errors, both MOE and pollster induced errors".
I knew someone would correct me on that and I really didn’t feel like doing the math. You are also correct that the results STILL don’t square with a 95% confidence level and their 4% confidence interval, even if I fudged a bit.
Table 2 is clearer in what it shows.
Yea, something doesn’t seem right to me either. I wonder if columns 2 and 3 are the actual results.
In the very first race you have the Republican winning in Wisconsin.
I'm not a "stats weenie" but I noticed that one too. In the first row, either one set of numbers are reversed or the "R" should be a "W."
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.