Posted on 09/05/2008 6:19:46 AM PDT by xzins
How Serious Is Polling's Cell-Only Problem?
The Landline-less Are Different and Their Numbers Are Growing Fast
Twenty years ago the survey research profession -- having grown comfortable with telephone interviewing as an alternative to personal interviewing for conducting surveys -- worried mostly about the roughly 7% of U.S. households that could not be interviewed because they had no telephone. Today our concern is somewhat different, and potentially more serious. According to government statistics released last month, nearly 13% of U.S. households (12.8%) cannot now be reached by the typical telephone survey because they have only a cell phone and no landline telephone.1
If people who can only be reached by cell phone were just like those with landlines, their absence from surveys would not create a problem for polling. But cell-only adults are very different. The National Health Interview Survey found them to be much younger, more likely to be African American or Hispanic, less likely to be married, and less likely to be a homeowner than adults with landline telephones. These demographic characteristics are correlated with a wide range of social and political behaviors.
Polling's cell phone problem is a new one. In early 2003, just 3.2% of households were cell-only. By the fall of 2004, pollsters and journalists were openly worrying about the potential bias that cell-only households might create for political surveys. The National Election Pool's exit poll found that 7.1% of those who voted on Election Day had only a cell phone, and these cell-only voters were somewhat more Democratic and liberal than those who said they had a landline telephone. But pre-election telephone polls in that election were generally accurate, and pollsters felt that they had dodged the proverbial bullet. This fortunate outcome was a result of the fact that the statistical weighting employed by most telephone polls helped to correct for the missing respondents. The fact that the cell-only group in 2004 was still a relatively small part of the overall population also helped mitigate the impact of the problem.
But given the speed with which the number of cell-only households has increased, there is growing concern within the polling business about how long the landline telephone survey will remain a viable data collection tool, at least by itself. At last month's annual meeting of the American Association for Public Opinion Research (AAPOR), survey research's top professional organization, an entire series of research panels focused on the cell phone issue. At one of the panels, a government researcher told the audience that the size of the cell-only group could approach 25% by the end of 2008 if the current rate of increase is sustained.
To monitor the impact of the cell-only phenomenon, the Pew Research Center conducted four studies in 2006 that included samples of cell phone numbers as well as a full sample of landline numbers.2 The four surveys covered a very wide range of topics, including use of technology, media consumption, political and social attitudes, and electoral engagement. Comparing the cell-only respondents with those reached on landlines allowed us to assess the degree to which our traditional surveys are biased by the absence of the cell-only respondents.
We compared the cell-only and landline respondents on 46 different survey questions. Across these questions, the average difference between cell-only and landline respondents was approximately 8 percentage points (7.8%), with the range of differences running from 0% (for a question about whether the respondent is "bored" by what goes on in Washington, DC) to 29% (being registered to vote). But the good news is that none of the measures would change by more than 2 percentage points when the cell-only respondents were blended into the landline sample. Thus, although cell-only respondents are different from landline respondents in important ways, they were neither numerous enough nor different enough on the questions we examined to produce a significant change in overall general population survey estimates when included with the landline samples and weighted according to U.S. Census parameters on basic demographic characteristics.
The picture is not entirely positive, however. While the cell-only problem is currently not biasing polls based on the entire population, it may very well be damaging estimates for certain subgroups in which the use of only a cell phone is more common. This concern is particularly relevant for young adults. According to the most recent government estimate, more than 25% of those under age 30 use only a cell phone. An analysis of young people ages 18-25 in one of the Pew polls found that the exclusion of the cell-only respondents resulted in significantly lower estimates of this age group's approval of alcohol consumption and marijuana use. Perhaps not surprisingly, excluding the cell-only respondents also yields lower estimates of technological sophistication. For example, the overall estimate for the proportion of 18-25 year olds using social networking sites is 57% when the cell-only sample is blended with the landline sample, while the estimate based only on the landline sample is 50%.
Including a cell-only sample with a traditional landline-based poll is feasible, as the four studies conducted last year indicate. But even if feasible, cell-only surveys are considerably more difficult and expensive to conduct than landline surveys. Federal law prohibits the use of automated dialing devices when calling cell phones so each number in the cell phone sample must be dialed manually. It also is common practice to provide respondents with a small monetary incentive to offset the cost of the airtime used during the interview. And the screening necessary to reach cell-only respondents among all of those reached on a cell phone greatly increases the effort needed to complete a given number of interviews. Pew estimates that interviewing a cell-only respondent costs approximately four to five times as much as a landline respondent.
Pollsters recognize that some type of accommodation for the cell-only population will have to be made eventually, as was clear from the large amount of research on the topic presented at the AAPOR conference last month. In addition to the use of so-called "dual frame samples" such as those described above (calling both a cell phone sample and a landline sample), practitioners are discussing other alternatives, including the establishment of panels of cell-only respondents that can be surveyed periodically to track their opinions, and employing mail or internet surveys to reach the cell-only population.
This article draws on research presented at the AAPOR conference, "What's Missing from National RDD Surveys? The Impact of the Growing Cell-Only Population," by Scott Keeter (Pew Research Center), Courtney Kennedy (University of Michigan and Pew Research Center), April Clark (Pew Research Center), Trevor Tompson (The Associated Press), and Mike Mokrzycki (The Associated Press).
Note
1Adding in the 2.2% of households with no phone service whatsoever, a total of 15.0% cannot be reached by landline surveys. The government report is by Stephen J. Blumberg and Julian V. Luke. "Wireless Substitution: Early Release of Estimates Based on the National Health Interview Survey, July – December 2006." Report by the U.S. Centers for Disease Control and Prevention, May 14, 2007.
2Details about the four studies can be found at the following links: "The Cell Phone Challenge to Survey Research"; "Online Papers Modestly Boost Newspaper Readership"; "A Portrait of Generation Next"; and "Cell-Only Voters Not Very Different."
Number of survey questions compared 46
Average (mean) difference between landline and cell-only samples across all 46 questions 7.8%
Range of differences (absolute value) 0% - 29%
Maximum change in final survey estimate when cell-only sample is blended in 2%
Average (mean) change in final survey estimate when cell-only sample is blended in 0.7%
There are other polling problems. I simply won’t respond to unknown phone calls. (See congressman billybob’s recent article in which he discusses what might have been a phony pollster)
Are there allowances for the large number of folks like me who won’t talk to them?
Are we a unique group with characteristics that would skew their results?
Nother fallacy is that they don’t call cell phone numbers. I changed from landlines to all cell last year, but I was able to keep my long-time phone number. I get the calls on my cell now. (I don’t take them, but I get them). I’m sure that many people do the same.
I won’t talk to them on landline or cell.
My boss, who is NOT a Republican, thinks that the majority of the cell-only crowd will vote for Obama. However, I don’t think it’s that cut-and-dried. I think a lot of the POTENTIAL Obama voters in the cell-only contingent won’t vote on election day.
Most won't even register.
I would say people screeening their phone calls with caller ID is likely to be a bigger problem than cell phones.
If I ever get polled, I will mislead them. I want the pollsters to look foolish. Maybe they will stop being considered news.
a lot of them will only vote when they can text message their vote in.
I don’t think Republicans are inclined to view favorably “folks” [they aren’t really humans]who: invade privacy; use property they didn’t pay for; take up valuable time that could be spent working or thinking or Freeping; and ask loaded questions like do you want your next president to be well educated (and then marking the yes as an Obama vote), all with an obvious political agenda.
In the event that I do talk to someone from a campaign I don’t give them much to go on.
When they call and ask what I want to see from their candidate I tell them I want to know the candidate’s opinions, not what I told him I wanted to hear.
Other polling problems ...
I got an enthusiastic call from an Obama support within the last week. She asked how I’d vote and I said I didn’t know enough about the local candidates. When she push on Obama I said that was a given and convinced her O had my vote.
I love those polls!
I’ll say this. All the single people I know don’t have land lines. Both sides. The Married ones do. It’s anecdotal at best but it’s what I have. (I’m 30)
However, I don’t think it’s that cut-and-dried. I think a lot of the POTENTIAL Obama voters in the cell-only contingent won’t vote on election day.
I tend to agree with your assessment. Also until the dynamics of the voters change to the extent that cell only voters become a larger impact then the current process will still be valid.
In a national election such as President they have this basically broken down to minute details and focus on very specific areas to determine or make projections. Example in PA the County were Scranton is located often determines the outcome of how the state of PA votes.... Thus they will spend more efforts to survey that county in detail to make their projections...
Excellent point. It adds to the likelihood that their numbers are skewed
Wow. You need to post that as its own thread.
I have a few telemarketers working for me.
Cell phones are great to call.
BUT... we try to stay away from prefixes that are used for the “prepaid minute” plans.
Those people don’t have money or jobs. Their phones are always getting turned off until they can “bump up” their minutes, etc.
Plus, we are seeing lots of people keeping their phone numbers after moving out of the area.
So if they are surveying people with a Michigan area code, they may be calling people who USED TO LIVE in the state but now live in Alabama.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.