Posted on 10/25/2012 6:47:37 PM PDT by GOPFlack
Part I: Real Pollsters Who Work for Campaigns
As we enter the homestretch of this election and we all hang by every thread of polling data, I thought I would share some insights into polling data that I have gleaned from being involved in elections for the past 26 years. Some of this content has been provided by smart posters here, and by other folks like Jay Cost at the Weekly Standard. But some content might surprise you and hopefully allow you to discern between good polls and bad ones, when to be concerned, when to be skeptical and above all what some of the quality pollsters do to get as close to the truth as possible and where even they admit they make mistakes.
First, let me lay out the difference between pollsters who want the truth and those who want to drive news cycles for media clients. Most of the best Democrat and GOP pollsters are people you have scantly heard about because their clients are not news outlets, but are instead campaigns, party committees and outside groups who need to target their spending and messaging. None of these firms do media polls. Rarely will they release their results publicly unless it is to counter a series of poorly done media polls at the client’s request. They are paid to be accurate and to tell the truth. Being on MSNBC or Fox is not how they make their living. It is no coincidence that most of these guys, Democrat or Republican generally come back with almost identical numbers when they go in the field at the same time. Why do you think the two POTUS campaigns schedule their stops in the same state? Both campaigns have roughly the same numbers on a nightly basis. Lying to your client could cost said pollster work down the road, especially if the sunshine pumping causes decisions that lead to a loss. An old pollster friend has told me many times in races – they (the Dems) see the same thing we do.
Private C&E pollsters have a few things in common that they believe in – no robo calls, live interviews only, a tight likely voter screen and crosstab questions to figure out where you are doing well or poorly, test messages, or to see where you have work to do. Certain pollsters that are also very familiar with certain states or regions have some things they will and won’t do in order to avoid quirks in particular areas they have picked up through years of experience. In addition, they prefer to use their own interviewers or a subcontractor with very good training for their interviewers – usually someone they have worked for years. You would be surprised how a poorly trained interviewer can get a bad batch of results in a hurry. Simply reading a screen as you enter data points is not foolproof if the interviewer is a flat out stupid, poorly trained or incompetent.
For example, I live in Virginia. Tuesday night a pollster called me on my landline. I didn’t immediately pick up on the name of the firm so I decided to answer to see if was in fact some sort of push poll. After the first two questions, I could tell it was a poll run by a campaign. Then, based on the lack of message testing, I could tell it was a tracking poll two questions later. They even threw Virgil Goode in there on the unaided ballot just to be on the safe side. At the end of the call, the interviewer disclosed that under Virginia law she was required to tell me that the poll was conducted by Romney for President. It was Romney’s Virginia track and it randomly pulled my number. I didn’t contaminate the sample, and they know this because I was required to answer who I voted for in the last election among other items. In the crosstabs, I will show up as a committed Republican, certain voter data point of which they will pull many when they do the breakdowns - just as Obama’s will find the same in his tracking. And both tracks will see some Democrats show up in the same manner. They expect it because it is a reality.
The interviewer was clearly well trained and was very clear spoken and professional. Contrast that interview with an interest group who called me with a poll a couple of weeks ago and the interviewer could barely speak English and had no clue what she was doing. Does anyone think that interviewer or pollster got much of any accurate data back? I’m highly doubtful.
A few other notes on private C&E pollsters – even before they track, or go in the field weekly in big races, they live by certain rules. One is never do a one night sample. The potential for contamination or a quirky result can be caused by any number of factors. Live interviews automatically reduce the sample size and simple higher standard deviation could skew your results. Most insist on two days if they aren’t tracking. Not only does it smooth out and reduce deviation, but it also can smooth out the case of a group fixated on a big sporting event, for example, that might actually tilt away or toward your candidate. That sounds silly, but you would be shocked at how little it takes to blow up a poll by 3 to 5 points in a hurry.
For example, a veteran GOP pollster told me once that he never polls the South or certain Midwestern states on Wednesday. The reason is because prayer groups and or Wednesday congregation will automatically cause an under sampling of his client’s core supporters. While that may not matter in California, it does matter in many states – including, by the way, Virginia from Fredericksburg South. To further make that point, Scott Rasmussen, who I have great respect for and I think does a great job given the technology he deploys (and I will discuss him in detail in the next section), often misses on state polls when he does a big batch on a single night in one state. I don’t know how many nightly calls he is making in Virginia on his normal swing state track. But obviously he took an extra heavy batch in Virginia on Wednesday for the results he released Thursday. He showed Romney up two among likely voters. Truth is he most likely under sampled Romney’s support by a couple because of the Southern Wednesday night anomaly. Romney has overwhelming rural and non - core DC metro support in areas where many churchgoers still meet on Wednesday night. Sample size can compensate here but not totally.
In addition, professional campaign pollsters do not, ever, reweight their results to anticipate some magical party ID change. They want to know who is voting, for whom they are voting, and why. That is why a poll done even in June will be at least a two night poll with a tight screen in order to reduce error and get a clear idea of what is really going on for the campaign. They don’t usually do registered voters unless to see if the base is motivated or demoralized and usually only very early in the campaign. If something moves in likely voter make up later they will ultimately pick it up quickly.
The takeaway – (other than beware of Wednesday samples in some states understating GOP support), this is how the guys who get paid for accuracy do it. Depending on your reaction, I will be happy to delve into Part II – bad media pollsters, hacks, and the good and bad on robo polling - including why Rasmussen is the only one of those you should pay any attention to.
Thanks for your indulgence. Hope this helps.
Please ping me to subsequent parts.
Thank you.
What published polls do you think are worth paying attention to?
Poll ping. Interesting read.
Please add me to the Part II list.
Great analysis! Would love a ping for part 2 :-)
Great analysis! Would love a ping for part 2 :-)
Good information, thank you. I look forward to reading part II.
Thanks a lot, and I’d love to read further installments. This is really appreciated and helpful.
I just had a trusting neophyte friend, who sincerely, but inaccurately, believes I’m knowledgeable about all things political ask me about whether she should be worried about the current polling numbers for Romney; does he have a real chance. I was able to give her a bit about over/under sampling, the trending vs spot numbers, comparisons to previous races, and so forth; but there’s too much I don’t know myself.
Rasmussen uses robo calls and his results are consistently at the top. In fact he was the most accurate in 2008 and I believe in 2004 as well.
Please ping me when you do part II. It helped me tremendously.
Should have added a request for pings to future installments. Thanks.
Put me on your ping list also, please! Thanky for your work here.
Ping request.
Thanks flack - non FR friends also thanking me for forwarding to them.
Very interesting, and please do add me to the ping list for Part II.
I should like to get pinged to Part II also. thanks.
Great post..... thanks for your effort
Great article.
Please add me to the part II ping list.
This is an outstanding post. Keep up the good work and like Mark Finkelstein you’ll be a rock star.
I’d like to read part 2, as well.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.