Skip to comments.Quality Control Among the Polls: Will RealClearPolitics.com Protect Its Brand?
Posted on 08/02/2012 5:08:39 AM PDT by Kaslin
RealClearPolitics.com was begun in 2000 and in the 12 years that have followed, it has become the gold standard for aggregators of political stories, punditry and polling data.
Almost every newscast of note refers to the RealClearPolitics average of polls, which the commentariat has come to rely upon as a way of smoothing out the inevitable differences between polls and the various methodologies they represent.
With success comes a problem, however, and that is the old rule of garbage in, garbage out.
Should RCP, as it is known among those who rely upon it, do something to guard its currency against devaluation?
Having built the best brand in the business, ought RCPs editors to take some role in preserving the quality of its product?
Heres the problem.
Any media organization can commission a poll. The quality of the results depends upon the professionalism of the pollsters conducting the survey.
Some polling organizations are so routinely lousy over a long enough period of time that they lose all credibility. Think Zogby. Ask yourself when you last saw a Zogby poll being cited as useful data?
Other polling efforts are kept afloat by big name sponsors despite long standing records of terrible bias, like the Minnesota Poll whichis simply an occasion of mirth among folks like the bloggers of Powerline whohave long tracked that polls hopelessly left-wing bias. The sponsorship of the Minneapolis Star Tribune keeps the Minnesota Poll going despite its transparent hackery.
PPP is a Democratic house which provides useful data if one corrects for the lie of the green.
Strategic Vision is thought to be biased in favor of conservatives, but I and some other conservatives view its results with some skepticism.
Rasmussen, by contrast, is generally thought by those on the center-right of the political spectrum to provide accurate results, as is Gallup.
So it goes. Everyone has opinions of pollsters, and RCP uses a blender and produces a broth that most folks find palatable.
Now, however, The New York Times/CBS/Quinnipiac poll of swing states raises a new question of whether obviously busted polls should be included in the RCP average.
In results released yesterday, this poll announced thatPresident Obama had comfortable leads in Pennsylvania, Ohio and Florida.
The most important of these results Florida and Ohioat first surprised and then either thrilled or discouraged observers depending on whether they were cheering for Team Romney or Team Obama.
Then, however, the partisan makeup of the respondents was revealed. (HotAir's Ed Morrissey is always quick to find and publish such key data.)
Turns out the sample Quinnipiac used in Florida had 9% more Democrats than Republicans.
The sample used in Ohio had 8% more Democrats than Republicans.
Not only did these samples fail to reflect the partisan make-up of the 2010 election turnout in either state, they even oversampled Democrats compared to the 2008 turnout and 2008 was the best year for Democratic turnout in a generation.
It is simply impossible to argue with a straight face that these results are at all useful as predictive of Novembers result or even of public opinion in either state as it exists today. But that is exactly what both the paper and the television network did. CBS opened its story by declaring that "President Obama leads Mitt Romney among likely voters in Ohio and Florida." The Times' headline asserted that "New Polls in Three Battleground States Show Obama Edge."
Interestingly, the Times' reporters, Jeff Zeleny and Dalia Sussman, avoided anything like CBS' declaration in their write-up, as though both were aware of the deep flaw in the polls' sample.
The New York Times and CBS are both biased media organizations the former greatly so, the latter less so but still left-of-center.
Everyone understands this, and arguments about this bias arent worth having.
Since these media organizations dont care about accuracy, neither will they care about a misleading poll carrying their brand name even though individual reporters might refuse through artful writing to pump a lousy poll.
But should RealClearPolitics.com care?
I interviewed Peter Brown of the Quinnipiac University Polling Institute on Wednesdays show. The transcript is here. Mr. Brown has no defense for the approach they took, other than to repeat that Quinnipiac has a great track record. I gave him opportunity after opportunity to defend the poll's methodology, and he simply could not answer the obvious questions.
When Mr. Brown agreed that a poll of all Democrats would have no utility in telling us anything worth knowing, the game was over, and he sensed it, refusing to provide any guidance on when a random sample crossed over from useless to useful.
Read the transcript and ask yourself why The New York Times or CBS or any news organization would use these results as the basis for a front page story?
There is no good reason to do so, no conclusions that can be drawn from the results about how the state is leaning in the fall or what issues are driving the electorate. If a poll of all Democrats is useless, then a poll wildly oversampling Democrats is also useless, though less obviously so.
Indeed, a sample that is ridiculously over-weighted with Democrats coming up with a result that President Obama is leading? That is worse than worthless. That is misleading.
So RCP has to decide whether to pollute its poll of polls with an obviously biased poll.
It took a dozen years to build a great brand. Will RCP act to protect it?
I have had the thought for several years now that other websites should create their own polling averages using whatever rules that they feel make the most accurate averages.
And then we can compare the poll aggregators in the same way we compare the polling firms, by seeing whose averages perform better over time as the most accurate.
I want to see threads that say something like: “In the 2014 election, the XYZ average performed the best, while the RCP and ZYX averages missed the mark.”
Poll averages are useless. They mix good polls, garbage polls and outliers taken with different methodologies at different times and give you an ‘average’. It’s useless, but journalists like it because they are lazy and don’t want to analyse individual polls, so they just talk about the ‘average.
The CBS/NYT poll mentioned in the article was actually very bad news for Obama. The same poll asked about senate races in OH, PA, and FL....the oversampled democrats preferred the Democrat candidates - by a larger margin than they preferred Obama over Romney.
Hussein is in trouble - that’s the real story that nobody reported.
So many Americans have come to loathe and detest pollsters that I suspect they are being lied to at least half the time.
The real news is always ignored by the LSM
I’m glad someone is finally discussing this topic. I’ve never liked the RCP average. How can you combine polls of “likely voters” with polls of “registered voters” and even “adults” and expect your average to reflect, much less predict, reality?
I am not to crazy about either