How Serious Is Polling’s Cell-Only Problem?

by Scott Keeter, Director, Survey Research, PewResearch Center

Twenty years ago the survey research profession — having grown comfortable with telephone interviewing as an alternative to personal interviewing for conducting surveys — worried mostly about the roughly 7% of U.S. households that could not be interviewed because they had no telephone. Today our concern is somewhat different, and potentially more serious. According to government statistics released last month, nearly 13% of U.S. households (12.8%) cannot now be reached by the typical telephone survey because they have only a cell phone and no landline telephone.¹

If people who can only be reached by cell phone were just like those with landlines, their absence from surveys would not create a problem for polling. But cell-only adults are very different. The National Health Interview Survey found them to be much younger, more likely to be African American or Hispanic, less likely to be married, and less likely to be a homeowner than adults with landline telephones. These demographic characteristics are correlated with a wide range of social and political behaviors.

Polling’s cell phone problem is a new one. In early 2003, just 3.2% of households were cell-only. By the fall of 2004, pollsters and journalists were openly worrying about the potential bias that cell-only households might create for political surveys. The National Election Pool’s exit poll found that 7.1% of those who voted on Election Day had only a cell phone, and these cell-only voters were somewhat more Democratic and liberal than those who said they had a landline telephone. But pre-election telephone polls in that election were generally accurate, and pollsters felt that they had dodged the proverbial bullet. This fortunate outcome was a result of the fact that the statistical weighting employed by most telephone polls helped to correct for the missing respondents. The fact that the cell-only group in 2004 was still a relatively small part of the overall population also helped mitigate the impact of the problem.

But given the speed with which the number of cell-only households has increased, there is growing concern within the polling business about how long the landline telephone survey will remain a viable data collection tool, at least by itself. At last month’s annual meeting of the American Association for Public Opinion Research (AAPOR), survey research’s top professional organization, an entire series of research panels focused on the cell phone issue. At one of the panels, a government researcher told the audience that the size of the cell-only group could approach 25% by the end of 2008 if the current rate of increase is sustained.

To monitor the impact of the cell-only phenomenon, the Pew Research Center conducted four studies in 2006 that included samples of cell phone numbers as well as a full sample of landline numbers.²The four surveys covered a very wide range of topics, including use of technology, media consumption, political and social attitudes, and electoral engagement. Comparing the cell-only respondents with those reached on landlines allowed us to assess the degree to which our traditional surveys are biased by the absence of the cell-only respondents.

We compared the cell-only and landline respondents on 46 different survey questions. Across these questions, the average difference between cell-only and landline respondents was approximately 8 percentage points (7.8%), with the range of differences running from 0% (for a question about whether the respondent is “bored” by what goes on in Washington, DC) to 29% (being registered to vote). But the good news is that none of the measures would change by more than 2 percentage points when the cell-only respondents were blended into the landline sample. Thus, although cell-only respondents are different from landline respondents in important ways, they were neither numerous enough nor different enough on the questions we examined to produce a significant change in overall general population survey estimates when included with the landline samples and weighted according to U.S. Census parameters on basic demographic characteristics.

The picture is not entirely positive, however. While the cell-only problem is currently not biasing polls based on the entire population, it may very well be damaging estimates for certain subgroups in which the use of only a cell phone is more common. This concern is particularly relevant for young adults. According to the most recent government estimate, more than 25% of those under age 30 use only a cell phone. An analysis of young people ages 18-25 in one of the Pew polls found that the exclusion of the cell-only respondents resulted in significantly lower estimates of this age group’s approval of alcohol consumption and marijuana use. Perhaps not surprisingly, excluding the cell-only respondents also yields lower estimates of technological sophistication. For example, the overall estimate for the proportion of 18-25 year olds using social networking sites is 57% when the cell-only sample is blended with the landline sample, while the estimate based only on the landline sample is 50%.

Including a cell-only sample with a traditional landline-based poll is feasible, as the four studies conducted last year indicate. But even if feasible, cell-only surveys are considerably more difficult and expensive to conduct than landline surveys. Federal law prohibits the use of automated dialing devices when calling cell phones so each number in the cell phone sample must be dialed manually. It also is common practice to provide respondents with a small monetary incentive to offset the cost of the airtime used during the interview. And the screening necessary to reach cell-only respondents among all of those reached on a cell phone greatly increases the effort needed to complete a given number of interviews. Pew estimates that interviewing a cell-only respondent costs approximately four to five times as much as a landline respondent.

Pollsters recognize that some type of accommodation for the cell-only population will have to be made eventually, as was clear from the large amount of research on the topic presented at the AAPOR conference last month. In addition to the use of so-called “dual frame samples” such as those described above (calling both a cell phone sample and a landline sample), practitioners are discussing other alternatives, including the establishment of panels of cell-only respondents that can be surveyed periodically to track their opinions, and employing mail or internet surveys to reach the cell-only population.

This article draws on research presented at the AAPOR conference, “What’s Missing from National RDD Surveys? The Impact of the Growing Cell-Only Population,” by Scott Keeter (Pew Research Center), Courtney Kennedy (University of Michigan and Pew Research Center), April Clark (Pew Research Center), Trevor Tompson (The Associated Press), and Mike Mokrzycki (The Associated Press).

Note

¹Adding in the 2.2% of households with no phone service whatsoever, a total of 15.0% cannot be reached by landline surveys. The government report is by Stephen J. Blumberg and Julian V. Luke. “Wireless Substitution: Early Release of Estimates Based on the National Health Interview Survey, July – December 2006.” Report by the U.S. Centers for Disease Control and Prevention, May 14, 2007.

²Details about the four studies can be found at the following links: “The Cell Phone Challenge to Survey Research“; “Online Papers Modestly Boost Newspaper Readership“; “A Portrait of Generation Next“; and “Cell-Only Voters Not Very Different.”