September 22, 2015

Who’s left out in a Web-only survey and how it affects results

Survey research is rapidly moving online – it’s cheaper, faster, provides greater flexibility in questionnaire design, and often has substantial advantages in data quality compared with phone surveys. Web surveys are being adopted in all sectors of the industry, from marketing to the U.S. Census Bureau’s American Community Survey to election polling.

That makes it increasingly important to assess the accuracy of these surveys. Surveys that include only those who use the internet (and are willing to take surveys online) run the risk of producing biased results. And, in fact, a notable share of Americans either cannot or will not complete a survey via the internet.

That raises two questions: Whom do you miss with a Web-only survey, and how does it affect your results?

In this instance, there was a way to get at the answers by turning to our American Trends Panel, a nationally representative group of Americans who have agreed to participate in our surveys. Most of the panel members participate via the Web, but a sizable number (representing nearly one-fifth of the public) do not. A little more than half of these non-Web participants are not online, and the rest would not provide an email address in order to be surveyed. However, we are able to survey the non-Web panel members by mail and assess how much, if at all, their non-participation would affect the outcome in a national poll conducted exclusively online.

Here are four key findings from our new report, “Coverage Error in Internet Surveys.” 

Most Survey Items Differ Little Between Full Sample and Web Sample1Survey results are mostly unaffected when the non-Web respondents are left out. There is little or no difference between the full (Web and mail combined) sample and Web-only sample in a majority of topics that were asked about. Only nine survey items we tested yielded a difference of 5 or more percentage points, while the other 397 (98%) of the survey items yielded a difference of 4 percentage points or less. More specifically, about two-thirds of the 406 items yielded estimates with a difference of zero or 1 point.

Selected Demographics of Respondents2Even though the people who couldn’t or wouldn’t take the Web survey were very different from the people who did, leaving them out didn’t make a difference in the national survey results. The non-Web respondents are more than twice as likely as the Web respondents to be ages 65 and older and nearly three times as likely to be black. Non-Web respondents have lower levels of education and family income than Web respondents. They are less likely than Web respondents to be married or partnered and more likely to be living alone. There are smaller differences between Web and non-Web respondents in terms of political identity and engagement.

Technology and Internet Items Show Sizeable Differences3One area where there were significant differences was survey items relating to technology. The nine out of 406 survey items that have a difference of at least 5 percentage points between the full sample and the Web sample are related to internet or technology use. There is a 13-point difference in daily internet use between the Web-only sample (82%) and of the total sample (69%). About eight-in-ten of Americans in the Web-only sample said they had sent email or texts to friends or family the day before, compared with about seven-in-ten in the total sample. Fully 90% in the Web-only sample own a desktop or laptop, compared with 80% in the total sample.

4Some findings among senior citizens, blacks and the less-educated showed large differences between the full sample and Web-only sample. Among adults 65 and older, the estimate of daily internet use is 35 points higher in a Web survey than the full sample (74% vs. 39%). But even some non-technology items show sizable differences. There is a 10-point difference among Americans ages 65 and older who are interested in Bible reading between the full sample and the Web-only sample. Blacks (65%) have a similar rate of employment as whites (61%) in the Web sample, but among the full sample the white employment rate is 8 points higher than among blacks (62% vs. 54%). Respondents with a high school education or less are 7 points more likely to have a driver’s license in the Web-only sample than those in the full sample.

Topics: American Trends Panel, Research Methods

  1. Photo of Eileen Patten

    is a research analyst focusing on Hispanic, social and demographic trends at Pew Research Center.

  2. is a research assistant focusing on internet, science and technology at Pew Research Center.

6 Comments

  1. Martin Vega11 months ago

    The issue of the representativeness of online vs. offline surveys in the case of the Hispanic population also involves additional and not easily understood caveats:
    1) The extent to which methodologies, design and sampling strategies should maintain internal consistency (i.e., oftentimes sample aggregation in online surveys draws quixotically from social media platforms, loyalty programs, single or double opt-in panels, and proprietary panels, often without due regard to employing screening criteria that should be Hispanic-relevant (i.e., self-identity, language, acculturation, etc.)
    2) Then there is the issue of wrongly applied metrics and scales regarding language, acculturation, etc., And, the acculturation modeling for Hispanics itself is done wrong
    3) Finally, there is no actual profiling of the recruited sample to match it (pre-field) against the Census, and, post-field the wrong weighting schemes are employed, such that the sample distributions of Hispanics by detailed demography or attributed behaviors are imprecise or inaccurate

  2. Sister Carrie11 months ago

    It’s discouraging that although I wanted to email this to a number of my friends, your processor does not accept this internet email addresses as valid

    wi.rr.com Time Warner in Wisconsin

    Please address the limitations on your email provisions for sharing articles. They’re too good to miss.

    1. David Kent11 months ago

      Thank you for your interest in Pew Research Center. To share the content of this post by email, please copy the following link into the text of a personal email message: pewrsr.ch/1KKZSg3

  3. Radioman KC11 months ago

    This was a horrible news story. And a horrible FB link to it.

    Don’t you have people who can write a coherent analysis? Isn’t the story here about elderly and undeducated results compared to everyone else?

    You missed the story. It was confusing and written by a data nerd, not a wordsmith.

  4. Barbara11 months ago

    If one only uses the internet, one is skewing the catholic research, since many practicing Catholics who attend weekly, and even daily mass, and follow the church’s teaching are among the poorest, and most disadvantaged. So leaving them out, definitely will give you skewed data. Now the spiritually lost who can afford a computer and internet access, who often will not find even an hour to go to mass and worship God, are representing the Catholic Church.
    That is just foolish and inaccurate. Glad to see the article. It certainly helps clarify things after seeing Catholic Vote and other polls.

    1. Eileen Patten11 months ago

      Hi Barbara,

      The full report that this blog post is based on actually contains a direct comparison of the non-web and web samples by religious affiliation. In fact, we find that there is no significant difference between the samples in the share that is Catholic (17% in the non-web sample and 19% in the web sample, before weighting). Catholics were the only religious group that was equally represented in the non-web and web samples. There is some more information about religious differences if you read the full report at: pewresearch.org/2015/09/22/cover…

      Thanks for your interest in our work.
      -Eileen