Poll Math (Fun with Zogby!)

10/21/2008 | PhilosopherStones

[PhilosopherStones]

Thought I'd have a little fun with poll math. Zogby was kind enough to provide some data which I have compiled into two tables below.

The original Zogby data (which they use to promote their business, so they must be proud of it) can be found here.

Explanation of Table 1:
Table one shows how accurate Zogby was when compared to actual election results (I've omitted races in which their was a significant third party vote or where Zogby had no polling data posted. Hey, you don't like it? Make yer own DANG spreadsheet!)

Col 1 is just the line number. Col 2 is the race. Col 3 is Zogby's pre-election number for Republicans. Col 4, ditto for Dems. Col 5 is the (absolute) spread between the two candidates according to Zogby. Col 6, 7 and 8 are the ACTUAL election results. Col 9 (E) is the difference between the Zogby results and the election results, ordered by greatest to least. Finally, Col 10 (C) shows whether Zogby called the race correctly (R) or incorrectly (W).

#	RACE	R	D	Dif	R	D	Dif	E	C
1	SEN, WI	38.0	47.2	9.2	67	30	37.0	27.8	R
2	GOV, TN	40.5	54.3	13.8	30	69	39.0	25.2	R
3	GOV, AZ	43.7	51.3	7.6	35	63	28.0	20.4	R
4	SEN, NM	37.5	58.2	20.7	30	70	40.0	19.3	R
5	GOV, OH	43.3	50.8	7.5	37	60	23.0	15.5	R
6	SEN, WA	46.1	50	3.9	39	58	19.0	15.1	R
7	GOV, CO	46.6	46.4	0.2	41	56	15.0	14.8	W
8	SEN, MN	42.8	50.5	7.7	38	58	20.0	12.3	R
9	HOUSE, IL-6	40.0	54.2	14.2	49	51	2.0	12.2	R
10	SEN, NY	33.0	57.3	24.3	31	67	36.0	11.7	R
11	GOV, AR	48.2	45.3	2.9	41	55	14.0	11.1	W
12	SEN, OH	47.3	48.7	1.4	44	56	12.0	10.6	R
13	HOUSE, VA-5	46.0	36.4	9.6	59	40	19.0	9.4	R
14	HOUSE, NM-1	43.9	53.2	9.3	50	50	0.0	9.3	W
15	SEN, PA	43.8	52.7	8.9	41	59	18.0	9.1	R
16	SEN, MI	41.9	49.3	7.4	41	57	16.0	8.6	R
17	SEN, MA	32.4	61.8	29.4	31	69	38.0	8.6	R
18	SEN, RI	38.8	53.4	14.6	47	53	6.0	8.6	R
19	GOV, CA	47.6	38.8	8.8	56	39	17.0	8.2	R
20	HOUSE, NY-20	51.2	37.1	14.1	47	53	6.0	8.1	W
21	GOV, GA	48.3	36.3	12.0	58	38	20.0	8.0	R
22	GOV, IA	46.4	47.6	1.2	45	54	9.0	7.8	R
23	GOV, WI	47.2	46.7	0.5	45	53	8.0	7.5	W
24	SEN, NV	58.3	36.8	21.5	55	41	14.0	7.5	R
25	GOV, OR	46.6	47.2	0.6	43	51	8.0	7.4	R
26	SEN, TX	54.8	35.9	18.9	62	36	26.0	7.1	R
27	GOV, PA	42.0	55.2	13.2	40	60	20.0	6.8	R
28	HOUSE, FL-22	39.6	49.2	9.6	51	47	4.0	5.6	R
29	HOUSE, VA-2	51.0	43.4	7.6	51	49	2.0	5.6	R
30	SEN, MD	44.3	48.8	4.5	44	54	10.0	5.5	R
31	GOV, CT	53.1	40.5	12.6	63	45	18.0	5.4	R
32	HOUSE, CO-7	39.8	54.1	14.3	42	51	9.0	5.3	R
33	HOUSE, CT-4	43.5	50.7	7.2	50	48	2.0	5.2	W
34	GOV, MI	42.7	51.6	8.9	42	56	14.0	5.1	R
35	HOUSE, KY-4	41.9	44.9	3.0	52	44	8.0	5.0	W
36	HOUSE, IN-2	39.0	51.9	12.9	46	54	8.0	4.9	R
37	GOV, MA	32.7	58.1	25.4	35	56	21.0	4.4	R
38	HOUSE, CT-2	46.6	42.3	4.3	50	50	0.0	4.3	R
39	GOV, NY	27.3	63.2	35.9	29	69	40.0	4.1	R
40	HOUSE, IA-1	41.5	48.6	7.1	43	54	11.0	3.9	R
41	HOUSE, OH-18	33.1	53.2	20.1	38	62	24.0	3.9	R
42	HOUSE, NC-11	43.3	47.6	4.3	46	54	8.0	3.7	R
43	SEN, FL	49.9	45.1	4.8	53	45	8.0	3.2	R
44	GOV, NV	46.6	47.5	0.9	48	44	4.0	3.1	W
45	HOUSE, PA-6	43.5	48.6	5.1	51	49	2.0	3.1	W
46	HOUSE, NY-26	47.0	46	1.0	52	48	4.0	3.0	R
47	HOUSE, NY-24	39.3	45.6	6.3	45	54	9.0	2.7	R
48	GOV, IL	33.0	39.6	6.6	40	49	9.0	2.4	R
49	HOUSE, MN-6	52.2	42.1	10.1	50	42	8.0	2.1	R
50	SEN, NJ	42.8	48.7	5.9	45	53	8.0	2.1	R
51	SEN, MT	45.5	47.4	1.9	49	49	0.0	1.9	R
52	SEN, TN	49.3	48.1	1.2	51	48	3.0	1.8	R
53	HOUSE, NY-25	45.0	48.6	3.6	51	49	2.0	1.6	W
54	GOV, MD	43.9	49.3	5.4	46	53	7.0	1.6	R
55	SEN, AZ	51.8	44.4	7.4	53	44	9.0	1.6	R
56	HOUSE, IN-9	46.0	47.5	1.5	46	49	3.0	1.5	R
57	HOUSE, KS-2	44.2	46.7	2.5	47	51	4.0	1.5	R
58	HOUSE, AZ-8	41.3	52.5	11.2	42	54	12.0	0.8	R
59	SEN, MO	47.3	48.6	1.3	47	49	2.0	0.7	R
60	GOV, MN	44.7	46.1	1.4	47	46	1.0	0.4	W
61	SEN, VA	43.7	44.8	1.1	49	50	1.0	0.1	R

You'll notice that we have to get all the way down to line 39 before we see that the actual results and the Zogby predictions are within 4 points of each other. In other words, more than 60 percent of the time, Zogby's predictions were outside of their own given margins of error.

The other thing to note is that even when Zogby was within the margin of error, they called the race wrong 18% of the time (4 out of 22).

Table 2 is the same as table 1 except ordered by Zogby spread:

#	RACE	R	D	Dif	R	D	Dif	E	C
1	GOV, CO	46.6	46.4	0.2	41	56	15.0	14.8	W
2	GOV, WI	47.2	46.7	0.5	45	53	8.0	7.5	W
3	GOV, OR	46.6	47.2	0.6	43	51	8.0	7.4	R
4	GOV, NV	46.6	47.5	0.9	48	44	4.0	3.1	W
5	HOUSE, NY-26	47.0	46	1.0	52	48	4.0	3.0	R
6	SEN, VA	43.7	44.8	1.1	49	50	1.0	0.1	R
7	SEN, TN	49.3	48.1	1.2	51	48	3.0	1.8	R
8	GOV, IA	46.4	47.6	1.2	45	54	9.0	7.8	R
9	SEN, MO	47.3	48.6	1.3	47	49	2.0	0.7	R
10	GOV, MN	44.7	46.1	1.4	47	46	1.0	0.4	W
11	SEN, OH	47.3	48.7	1.4	44	56	12.0	10.6	R
12	HOUSE, IN-9	46.0	47.5	1.5	46	49	3.0	1.5	R
13	SEN, MT	45.5	47.4	1.9	49	49	0.0	1.9	R
14	HOUSE, KS-2	44.2	46.7	2.5	47	51	4.0	1.5	R
15	GOV, AR	48.2	45.3	2.9	41	55	14.0	11.1	W
16	HOUSE, KY-4	41.9	44.9	3.0	52	44	8.0	5.0	W
17	HOUSE, NY-25	45.0	48.6	3.6	51	49	2.0	1.6	W
18	SEN, WA	46.1	50	3.9	39	58	19.0	15.1	R
19	HOUSE, CT-2	46.6	42.3	4.3	50	50	0.0	4.3	R
20	HOUSE, NC-11	43.3	47.6	4.3	46	54	8.0	3.7	R
21	SEN, MD	44.3	48.8	4.5	44	54	10.0	5.5	R
22	SEN, FL	49.9	45.1	4.8	53	45	8.0	3.2	R
23	HOUSE, PA-6	43.5	48.6	5.1	51	49	2.0	3.1	W
24	GOV, MD	43.9	49.3	5.4	46	53	7.0	1.6	R
25	SEN, NJ	42.8	48.7	5.9	45	53	8.0	2.1	R
26	HOUSE, NY-24	39.3	45.6	6.3	45	54	9.0	2.7	R
27	GOV, IL	33.0	39.6	6.6	40	49	9.0	2.4	R
28	HOUSE, IA-1	41.5	48.6	7.1	43	54	11.0	3.9	R
29	HOUSE, CT-4	43.5	50.7	7.2	50	48	2.0	5.2	W
30	SEN, MI	41.9	49.3	7.4	41	57	16.0	8.6	R
31	SEN, AZ	51.8	44.4	7.4	53	44	9.0	1.6	R
32	GOV, OH	43.3	50.8	7.5	37	60	23.0	15.5	R
33	GOV, AZ	43.7	51.3	7.6	35	63	28.0	20.4	R
34	HOUSE, VA-2	51.0	43.4	7.6	51	49	2.0	5.6	R
35	SEN, MN	42.8	50.5	7.7	38	58	20.0	12.3	R
36	GOV, CA	47.6	38.8	8.8	56	39	17.0	8.2	R
37	GOV, MI	42.7	51.6	8.9	42	56	14.0	5.1	R
38	SEN, PA	43.8	52.7	8.9	41	59	18.0	9.1	R
39	SEN, WI	38.0	47.2	9.2	67	30	37.0	27.8	R
40	HOUSE, NM-1	43.9	53.2	9.3	50	50	0.0	9.3	W
41	HOUSE, FL-22	39.6	49.2	9.6	51	47	4.0	5.6	R
42	HOUSE, VA-5	46.0	36.4	9.6	59	40	19.0	9.4	R
43	HOUSE, MN-6	52.2	42.1	10.1	50	42	8.0	2.1	R
44	HOUSE, AZ-8	41.3	52.5	11.2	42	54	12.0	0.8	R
45	GOV, GA	48.3	36.3	12.0	58	38	20.0	8.0	R
46	GOV, CT	53.1	40.5	12.6	63	45	18.0	5.4	R
47	HOUSE, IN-2	39.0	51.9	12.9	46	54	8.0	4.9	R
48	GOV, PA	42.0	55.2	13.2	40	60	20.0	6.8	R
49	GOV, TN	40.5	54.3	13.8	30	69	39.0	25.2	R
50	HOUSE, NY-20	51.2	37.1	14.1	47	53	6.0	8.1	W
51	HOUSE, IL-6	40.0	54.2	14.2	49	51	2.0	12.2	R
52	HOUSE, CO-7	39.8	54.1	14.3	42	51	9.0	5.3	R
53	SEN, RI	38.8	53.4	14.6	47	53	6.0	8.6	R
54	SEN, TX	54.8	35.9	18.9	62	36	26.0	7.1	R
55	HOUSE, OH-18	33.1	53.2	20.1	38	62	24.0	3.9	R
56	SEN, NM	37.5	58.2	20.7	30	70	40.0	19.3	R
57	SEN, NV	58.3	36.8	21.5	55	41	14.0	7.5	R
58	SEN, NY	33.0	57.3	24.3	31	67	36.0	11.7	R
59	GOV, MA	32.7	58.1	25.4	35	56	21.0	4.4	R
60	SEN, MA	32.4	61.8	29.4	31	69	38.0	8.6	R
61	GOV, NY	27.3	63.2	35.9	29	69	40.0	4.1	R

Here, look at the races that Zogby found close. Particularly, look at where Zogby had the race closer than 4 points. You'll find that they called the wrong winner a whopping 39% of the time (7/18 races).

Don't let the pollsters get you down. This thing is winnable!

[PhilosopherStones]

I still have the spreadsheet if any stats wienies want to make a point...

Otherwise, as always, feel free to revise and extend my remarks.

[TonyInOhio]

Great work! Thanks for taking the time to sort out the data. This is very helpful for the poll junkies.

[gleeaikin]

election poles ping

[gleeaikin]

election poles ping

[USFRIENDINVICTORIA]

Very interesting analysis. I just have a couple of formatting suggestions. It would probably be clearer if you didn't use "R" for "correct" -- perhaps use a "C".

Also, if you used subscripts to distinguish between the "predicted" and "actual" results. E.g. R_p and R_a.

[TruthHound]

Just scanning the numbers, it seems the overriding trend with Z is that he gives more of an an advantage to the Republicans than the actual vote reflected—which is not encouraging.

[PhilosopherStones]

Too late now. Next time I’ll FReepmail you for suggestions before I post any formatted text.

[Thane_Banquo]

And Zogby under-estimates Republicans by an average of 4 pts. Surprise! Surprise!

[softwarecreator]

Over at "http://www.electoral-vote.com/" they have the EV at Obama 364, McCain 171 and Ties 3.

Anyone know just how accurate this site is? It seems way out-of-wack to me.

[bigbob]

Very nice!

I’ve been hoping to come across a time series of 2008 Presidential poll data that includes error bands - let’s just be generous and say 2 sigma, although I’d really like to see 3 sigma. My gut feel is there would be no significant trend.

[5thGenTexan]

It's (R)ight vs. (W)rong on the Correctness column.

[scratcher]

Looks like just the opposite. So it’s good news.

[USFRIENDINVICTORIA]

Feel free -- although those are the only things I could think of. It always seems that the "best" format only reveals itself after publication.

[USFRIENDINVICTORIA]

My point was that the (R)ight is too easy to confuse with (R)epublican. According to Murphy's law, some people will be confused, whenever they can be.

[TruthHound]

Ummm, by my count (maybe I’m not reading the chart right), the races where Z gave the Republican MORE percentage in his projection than the actual vote were 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 15, 16, 17, 20, 22, 23, 24, 25, 27, 30, 34...need I go on?

[drangundsturm]

Once again I'll point out that the MOE is almost always misinterpreted by the media. If a poll has an MOE of 4 points, that means that both the democrat and gop estimates are withing 4 points of the polled number 95% of the time. That means that the *difference* between the two numbers can be up to 8 points off. Under those circumstances, a candidate would have to be up by 9 full points to be "outside the margin of error".

I even see Rasmussen himself in interviews misstating this simple fact. I've seen him say things like "Obama is up 5 which is outside our margin of error of 3". Nope, he would have to be up by 7 to be outside the margin of error of 3, because MOE does not measure differences between the numbers, but rather applies to EACH of the candidates's numbers.

So, when you point out that you have to go down to column 39 to see actual results within the MOE of 4, what you should really be saying is that you have go down to column 21 before you find results that exceed the combined candidate MOE of 8.

Still, your point is well taken, 21 out of 61 races were outside the doubled MOE. In theory only 5% should have been outside that figure not 33%.

This can easily be explained though, by the second of the great polling fallacies. MOE only measures "statistical" error. That is error due solely to random sampling chance. There are many other kinds of errors that occur in polls and they are additive to MOE. Those range from 1 to 6 percent *in addition to MOE*, typically 2-4% when studied on major pollsters, far more on less reliable pollsters.

Applying these other sources of error, we find that in this set of polls Zogby had really a fully loaded error of 9.5 points per candidate, or 19 points overall. Only with that figure do we get to the 95% confidence level (95% of the races listed fell within that level of error).

In this set of polls, then, a candidate had to be up by 20 points to be "outside the margin of all errors, both MOE and pollster induced errors".

[PhilosopherStones]

I knew someone would correct me on that and I really didn’t feel like doing the math. You are also correct that the results STILL don’t square with a 95% confidence level and their 4% confidence interval, even if I fudged a bit.

Table 2 is clearer in what it shows.

[hudsonohio]

Yea, something doesn’t seem right to me either. I wonder if columns 2 and 3 are the actual results.

[NavVet]

In the very first race you have the Republican winning in Wisconsin.

[Tarantulas]

In the very first race you have the Republican winning in Wisconsin.

I'm not a "stats weenie" but I noticed that one too. In the first row, either one set of numbers are reversed or the "R" should be a "W."