Scott Keeter, Director of Survey Research, Pew Research Center
The Pew Research Center recently released an instant poll (see it here) gauging public reaction to the outcome of the presidential election. Unlike the bulk of the Pew Research Center’s survey research, which is based on traditional telephone surveys, this poll was conducted online using Google Consumer Surveys. Director of Survey Research Scott Keeter describes the recent project.
Why did Pew Research work with Google to conduct this survey?
Scott Keeter: Technological advancements are providing new ways of conducting public opinion and social research, and the Pew Research Center is well situated to help the survey research profession explore and evaluate some of these new methods. Google has developed an alternative method of data collection that produces results that appear to track well with what we find using traditional surveys. To gain a better understanding of the potential for this method, we undertook an extensive evaluation of the comparability of their results and the results of similar questions from our telephone surveys. On the basis of this evaluation we decided to join forces to conduct certain kinds of surveys, such as immediate reactions to events like the outcome of the election.
How was the survey with Google conducted?
Google Consumer Surveys is a new tool developed by Google that interviews a sample of internet users from a diverse group of about 80 publisher sites that allow Google to ask one or two questions of selected visitors as they seek to view content on the site. It does not constitute a probability sample of all internet users, but it is also not an “opt in” or panel survey, which is the case for many online surveys. There is a lot more detail about the methodology in the analysis on our website.
How is this different from other national public opinion surveys conducted by the Pew Research Center?
Most of our surveys of the public are conducted by telephone using random-digit-dialing of landlines and cell phones. This method allows us to reach nearly 100% of U.S. households and provide the vast majority of the population with a chance to be included. The Google method surveys only internet users, though the internet population now constitutes 85% of the adult population. To the extent that the Google method accurately represents the views of internet users, results based on internet users and those based on the general public won’t usually differ very much. Another difference is that telephone surveys are administered by an interviewer, with respondents hearing the questions and answering. Google surveys are online, where respondents complete the questions themselves, and that difference in the mode of administration is known to produce certain kinds of differences in results.
How accurate are the results?
We are continuing to explore this question, but over the course of six months from May to October, 2012, we compared results for more than 40 questions asked in dual frame telephone surveys to those obtained using Google Consumer Surveys. Questions across a variety of subject areas were tested, including demographic characteristics, technology use, political attitudes and behavior and civic engagement. Across these various types of questions, the median difference between 48 results obtained from Pew Research surveys and using Google Consumer Surveys was 3 percentage points. The mean difference was 6 points.
Is this kind of survey a replacement for the traditional phone survey?
No. One of the most important limitations of the Google method is the inability to ask more than two questions of any single respondent. That means that we are not able to do in-depth analyses of the data, nor can we take advantage of multiple measures to help us understand why people believe or do what they do. But the Google method does have certain important advantages, including the ability to gather data quickly. For this reason we think it is likely to be useful for quick reaction polling.
Despite the challenges we face, the telephone survey method still works well. The Google method and others under development will likely be an important addition to the research tool kit available to pollsters.
…
