While the Pew Research Center, CNN/ORC and Washington Post/ABC News pollsters all took a similar approach in asking whether people felt the Department of Justice was right or wrong to subpoena the Associated Press reporters’ phone records, there were multiple differences in the phrasing, structure and context of the questions that help to explain the different findings.

Comparing Three Questions

Let’s start with the Pew Research Center’s survey, which found more disapproval than approval of the DOJ’s actions by a slim 44%-36% margin. Earlier in the survey, Pew Research asked how closely people had followed the news story. Barely a third reported following it very closely (16%) or fairly closely (19%), while most (64%) said they had not followed too closely or at all.

Pew Research then went on to ask if respondents approved or disapproved of the Justice Department’s decision to subpoena the phone records of AP journalists as part of an investigation into the disclosure of classified information and found slightly more disapproving than approving.

By comparison, the Washington Post/ABC News survey described the situation as follows: The AP reported classified information about U.S. anti-terrorism efforts and prosecutors have obtained AP’s phone records through a court order. Do you think this action by federal prosecutors is or is not justified? The survey found substantially more saying the actions were justified, by a 52%-33% margin. The difference is likely attributable to three differences in the way the surveys described the situation. First, Pew Research referred to the decision to subpoena phone records while the WP/ABC survey said the records were obtained… through a court order, which may make the actions seem more legitimate by highlighting a court’s involvement. Second, what Pew Research described as classified information the WP/ABC poll described as classified information about U.S. anti-terrorism efforts; past research suggests the public is willing to accept broader government actions in the defense against terrorism. Third, Pew Research mentioned the Justice Department, while WP/ABC mentioned federal prosecutors. It is likely that the DOJ, as a more political institution associated with the administration, is viewed less favorably than are prosecutors. All three of these differences likely contribute to the remarkably different survey results.

A third CNN/ORC survey conducted Friday and Saturday (the other two surveys were in the field Thursday through Sunday) used different language as well. Their line of questioning mentions that it was classified information about U.S. anti-terrorism efforts, as did the WP/ABC question. It also mentioned that it was the Justice Department, as did Pew Research. But CNN/ORC differed from the other polls in saying the phone records were secretly collected, rather than just subpoenaed or obtained through a court order. All in all, this question found more saying the DOJ’s actions were unacceptable than acceptable by a 52% to 43% margin, similar to the balance of opinion in the Pew Research survey.

There are two other differences that stand out in the CNN/ORC survey. First, prior to asking about whether the actions were acceptable, respondents were first asked how important the issue was, with the vast majority saying it was very important (53%) or somewhat important (34%), and few saying not too (8%) or not at all (4%) important. In this context, far more respondents offered an opinion when then probed about the appropriateness of the DOJ’s actions – just 5% offered no opinion on the CNN/ORC survey, compared with 15% on the WP/ABC survey, and 20% on the Pew Research survey.

Moreover, the CNN/ORC survey is the one out of these three that found a substantial difference of opinion along partisan lines: Republicans were about half as likely as Democrats to say the DOJ’s actions were acceptable (30% vs. 57%). In the Pew Research Poll, there was only a slight partisan difference (36% of Republicans and 43% of Democrats approved of the DOJ’s actions), and there was virtually no difference in the WP/ABC poll (54% of Democrats and 51% of Republicans said the actions were justified).

A Later Entry

Late in the day on Tuesday, Fox News released yet a fourth survey question along the same lines, with the most lopsided balance of opinion of the questioning so far. It is possible that opinions shifted over time, and the Fox News survey began two days later and ended one day later than those conducted by Pew Research and WP/ABC. It’s also worth noting that Fox News surveys are restricted to registered voters rather than the general public, a narrower pool that is generally more Republican in its political leanings.

But it is also likely that the language and context of the question are substantial factors. Where the first three polls said the DOJ subpoenaed, obtained through a court order, or secretly collected the phone records, the Fox News question used the phrase secretly seized to describe the DOJ’s actions. The Fox News survey also pointed out that the records were extensive, included both work and personal phones, and that they did so without giving the news organization prior notice, as is customary. By two-to-one (60% vs. 31%) more said this action by the government went too far rather than was probably justified.

An additional factor that likely affected the responses to the Fox News question was the context in which it was presented. Immediately prior to asking about the legitimacy of the DOJ’s actions, Fox News asked: Does it feel like the federal government has gotten out of control and is threatening the basic civil liberties of Americans, or doesn’t it feel this way to you? There is a long track record in the survey industry of how framing a question in a particular context can affect responses, and there is little doubt that raising this broad criticism of government – which roughly two-thirds (68%) of respondents agreed with – prior to the DOJ question had some effect.

The Lesson?

In an ideal world, understanding the impact of question wording differences on surveys is straightforward – respondents are randomly assigned to groups, each of which is asked the same question in the same context with only the language of interest modified. But in the real world, it’s rarely so simple. In the above examples, each polling organization made good-faith efforts to describe the facts of the situation as accurately as possible, but the word choices and context make it impossible to identify one particular phrase or concept that tipped the public’s thinking.

The fact that the language and context made a difference in how respondents reacted to the questions is a challenge of polling in a low-information situation. In this case, the DOJ news story was quickly developing, some facts were still unclear (such as how the department obtained the phone records), and the subject matter was somewhat complex. As the Pew Research survey showed, most Americans paid little or no attention to the story through Sunday – far less than was paid to the Benghazi and IRS news coverage. The Pew Research survey found that those who paid at least a fair amount of attention  took a far more negative view of the Justice Department’s actions. Sometimes the views of attentive respondents can be a leading indicator of the direction of public opinion if and when a story grows. But sometimes the pattern of responses is more partisan. In this case, it was the more attentive Republicans who expressed particular disapproval of the DOJ’s actions, which may reflect that they were simply more attuned to the political implications of the scandal for the Obama administration.

Overall, while wide differences across polls suggest that public reactions may be only loosely formed, there is important information to glean from situations like these. There is likely a core of Americans who support the DOJ’s actions irrespective of the survey language (the 31% who say justified even in the Fox News survey). And another core who concretely oppose it (the 33% who say not justified in the WP/ABC survey). And for the rest, there are clues as to the types of considerations that may affect their views if and when attention grows and information becomes more widespread.

The Pew Research Center recently released an instant poll (see it here) gauging public reaction to the outcome of the presidential election. Unlike the bulk of the Pew Research Center’s survey research, which is based on traditional telephone surveys, this poll was conducted online using Google Consumer Surveys. Director of Survey Research Scott Keeter describes the recent project.

Why did Pew Research work with Google to conduct this survey?

Scott Keeter: Technological advancements are providing new ways of conducting public opinion and social research, and the Pew Research Center is well situated to help the survey research profession explore and evaluate some of these new methods. Google has developed an alternative method of data collection that produces results that appear to track well with what we find using traditional surveys. To gain a better understanding of the potential for this method, we undertook an extensive evaluation of the comparability of their results and the results of similar questions from our telephone surveys. On the basis of this evaluation we decided to join forces to conduct certain kinds of surveys, such as immediate reactions to events like the outcome of the election.

How was the survey with Google conducted?

Google Consumer Surveys is a new tool developed by Google that interviews a sample of internet users from a diverse group of about 80 publisher sites that allow Google to ask one or two questions of selected visitors as they seek to view content on the site. It does not constitute a probability sample of all internet users, but it is also not an “opt in” or panel survey, which is the case for many online surveys. There is a lot more detail about the methodology in the analysis on our website.

How is this different from other national public opinion surveys conducted by the Pew Research Center?

Most of our surveys of the public are conducted by telephone using random-digit-dialing of landlines and cell phones. This method allows us to reach nearly 100% of U.S. households and provide the vast majority of the population with a chance to be included. The Google method surveys only internet users, though the internet population now constitutes 85% of the adult population. To the extent that the Google method accurately represents the views of internet users, results based on internet users and those based on the general public won’t usually differ very much. Another difference is that telephone surveys are administered by an interviewer, with respondents hearing the questions and answering. Google surveys are online, where respondents complete the questions themselves, and that difference in the mode of administration is known to produce certain kinds of differences in results.

How accurate are the results?

We are continuing to explore this question, but over the course of six months from May to October, 2012, we compared results for more than 40 questions asked in dual frame telephone surveys to those obtained using Google Consumer Surveys. Questions across a variety of subject areas were tested, including demographic characteristics, technology use, political attitudes and behavior and civic engagement. Across these various types of questions, the median difference between 48 results obtained from Pew Research surveys and using Google Consumer Surveys was 3 percentage points. The mean difference was 6 points.

Is this kind of survey a replacement for the traditional phone survey?

No. One of the most important limitations of the Google method is the inability to ask more than two questions of any single respondent. That means that we are not able to do in-depth analyses of the data, nor can we take advantage of multiple measures to help us understand why people believe or do what they do. But the Google method does have certain important advantages, including the ability to gather data quickly. For this reason we think it is likely to be useful for quick reaction polling.

Despite the challenges we face, the telephone survey method still works well. The Google method and others under development will likely be an important addition to the research tool kit available to pollsters.

…

Q. What is the value of polls that match an incumbent or specific candidate against a “generic” Republican or Democrat? Some of these polls give respondents an extra option by choosing the answer “it depends on the candidate,” and some just leave as the only other choice being “undecided” or expressing “no opinion.” I assume that makes a difference.

A. The so-called “generic ballot” provides a snapshot of how an incumbent president would fare against an unnamed challenger from the opposing party. In this regard, the question is simultaneously posing a referendum on the incumbent and a test of the popularity of the other party. The trend on this question can be informative. In Pew Research Center polls in July and August, Barack Obama ran about even with an unnamed Republican; he held a significant lead earlier in the year. At the same time, however, the generic ballot has no predictive power, particularly at this very early stage in the presidential race.

Generic questions that invite respondents to say “it depends” will certainly get a much higher percentage declining to choose between the candidates. That may be a more honest reading of public opinion at this point in the campaign, but is even further from simulating the actual choice that voters will face in November 2012.

The so-called “generic ballot” tends to be a good predictor of the outcome of congressional elections in the off-years. Many survey organizations, including Pew Research Center, use such questions to help gauge the size of possible swings in party representation in Congress. But because presidential elections ultimately hinge on a much more personal choice, with far more information available to the average voter, generic questions are rarely used once the field of nominees has been winnowed.

In every campaign cycle, pollwatchers pay close attention to the details of every election survey. And well they should. But focusing on the partisan balance of surveys is, in almost every circumstance, the wrong place to look.

The latest Pew Research Center survey conducted July 16-26 among 1,956 registered voters nationwide found 51% supporting Barack Obama and 41% Mitt Romney. This is unquestionably a good poll for Obama – one of his widest leads of the year according to our surveys, though largely unchanged from earlier in July and consistent with polling over the course of this year.

And the survey did interview more Democrats than Republicans; 38% of registered voters said they think of themselves as Democrats, 25% as Republicans, and 33% as independents (to clarify, some reporters and bloggers incorrectly posted their own calculations of party identification based on unweighted figures). That’s slightly more Democrats than average over the past year, and slightly fewer Republicans. Recent Pew Research Center surveys have found anywhere from a one-point to a ten-point Democratic identification advantage, with an average of about seven points.

While it would be easy to standardize the distribution of Democrats, Republicans and independents across all of these surveys, this would unquestionably be the wrong thing to do. While all of our surveys are statistically adjusted to represent the proper proportion of Americans in different regions of the country; younger and older Americans; whites, African Americans and Hispanics; and even the correct share of adults who rely on cell phones as opposed to landline phones, these are all known, and relatively stable, characteristics of the population that can be verified off of U.S. Census Bureau data or other high quality government data sources.

Party identification is another thing entirely. Most fundamentally, it is an attitude, not a demographic. To put it simply, party identification is one of the aspects of public opinion that our surveys are trying to measure, not something that we know ahead of time like the share of adults who are African American, female, or who live in the South. Particularly in an election cycle, the balance of party identification in surveys will ebb and flow with candidate fortunes, as it should, since the candidates themselves are the defining figureheads of those partisan labels. Thus there is no timely, independent measure of the partisan balance that polls could use for a baseline adjustment.

These shifts in party identification are essential to understanding the dynamics of American politics. In the months after the Sept. 11 terrorist attacks, polls registered a substantial increase in the share of Americans calling themselves Republican. We saw similar shifts in the balance of party identification as the War in Iraq went on, and in the build-up to the Republicans’ 2010 midterm election victory. In all of those instances, had we tried to standardize the balance of party identification in our surveys to some prior levels, our surveys would have fundamentally missed what were significant changes in public opinion.

Q. Each year, the Pew Research Center’s Global Attitudes Project conducts public opinion polls around the world. Isn’t it difficult to poll in some countries? And how confident are you in the poll findings?

A: There is no question that the process of fielding surveys in foreign countries can be challenging and even difficult, especially as the majority of our polls are conducted through face-to-face interviews. That said, the rigorous quality standards we apply to our Pew Global Attitudes survey work abroad are the same as those we apply here in the U.S. Namely, we strive to ensure that our surveys accurately represent the populations of individual countries, and that we ask questions that deliver valid information about how people view important events and issues of the day.

The task of ensuring our surveys represent the adult-age population a foreign country requires employing rigorous sampling methods. The first step is to work closely with our principal research partners to identify local polling firms with knowledgeable staff and proven track-records in designing large survey samples. The good news is that political and economic changes over the past two decades have generally increased the demand for social and market research and have made it easier for us to find local, capable research firms.

The not-so-good news is that, compared with the U.S., the penetration of landline phones in many countries has not gotten to a point at which we have felt comfortable fielding national surveys by phone. And while mobile phones are quickly becoming an integral part of life for many people around the globe, reliable information about mobile users and how to integrate them into national samples remains limited. Thus, with the exception of a few countries such as Britain, France, Germany and Japan, we have erred on the side of reliability and have administered our surveys in person, rather than by phone. This means that unlike national surveys in the U.S., which can be completed in just days, most Global Attitudes surveys take two or more weeks to complete.

Due to their use of proven sampling techniques, the local vendors we work with can achieve nationally representative surveys by conducting face-to-face surveys with about 1,000 respondents. The key is ensuring that these respondents are selected in a random, unbiased fashion – and that all adult members of a country’s population are eligible for inclusion in a survey. More often than not, we meet these requirements by using multi-stage, cluster samples.

What this means is that rather than randomly selecting individuals directly (by phone, for example), we first randomly select clusters of individuals – beginning with relatively large territorial units, akin to counties in the U.S. Once these primary clusters are selected, we randomly select smaller territorial units, until we work our way down to city blocks or villages. At this stage, interviewers either visit addresses selected randomly from a list, or they follow a so-called “random walk” in which they visit every third or fourth residence along a set route. At each residence, interviewers randomly select a respondent by using a Kish grid (a detailed list of all household members) or by selecting the adult who has had the most recent birthday.

As mentioned above, in most countries surveyed by Pew Global Attitudes, multi-stage, cluster samples are used to field nationally representative surveys. However, in a few instances we are unable to conduct full, national surveys face-to-face. Sometimes, the limiting factors are cost and time. In China, for example, it would take many weeks to collect face-to-face interviews from across the country, and it would be prohibitively expensive to transport trained interviewers long distances. Therefore, for now our surveys represent only 57% of the Chinese population (those mostly in urban areas). We hope to expand our coverage in the future.

In other instances, concern for the safety of interviewers keeps us from fielding truly national surveys. In Pakistan, for example, our surveys exclude 15% of the population due to concerns about sending interviewers into frequently violent border regions. As in China, our survey respondent base is predominantly urban because of this. But regardless of the scope, our surveys in China and Pakistan are held to the same methodological standards as our surveys conducted elsewhere.

One of the quality checks we perform for all Pew Global Attitudes surveys is to compare the demographic characteristics of the people included in our surveys with census or other official data that describe the gender, age or educational makeup of the population. In cases where our data departs more than a few percentage points from official statistics, we may decide to adjust our data through the mathematical procedure of weighting. Weighting is a common practice in survey research. Properly applied, it can improve our ability to accurately report how prevalent or variable attitudes are in a given society.

For all our surveys, of course, we calculate country-specific margins of sampling error, based on the number of people surveyed and whether the survey was based on random dialing by phone or a multi-stage, cluster sample. These margins of error are integral to our ability to identify attitudinal shifts or statistically-significant differences. Over the years, our key trends have proven highly reliable; they have moved in directions that track well with political and economic developments or have remained relatively stable in the absence of major events or changes at the local, regional or global level. This is another reason we stand behind the accuracy of our data.

Beyond the technical aspects of sample design, our confidence in Pew Global Attitudes’ findings stems from the time and effort we put into the design and translation of our questionnaires. In a given year, our survey questions may be translated into twenty or more languages. Coordinating so many translations can be daunting. Fortunately, since the Pew Global Attitudes Project began in 2002, we have assiduously archived translated versions of the annual questionnaire. As we develop each new round of the survey, we are able to draw on these tried-and-tested translations when we repeat specific items or trends.

But past translations are of little help when it comes to new questions or new countries. In such instances, we rely on local polling firms to first translate our questions into the appropriate local language or languages. As a standard operating procedure, we then have the translation re-translated into English by a bilingual person who has not seen the original English-language questionnaire. This is called back-translation, and it is highly useful for identifying questions that have been poorly or incorrectly translated. For complex questions or especially challenging languages, we often take the additional step of consulting with linguistic experts to perfect translation of new items.

Regardless of language, the goal of the translation process is always the same: to ensure that we ask questions that reflect our intended meaning, with results that can be compared cross-nationally.

A final reason for confidence in our findings is the careful training of interviewers and close supervision of fieldwork. In each country, prior to fieldwork, local research firms train their interviewers to properly administer the questionnaire. This includes briefing interviewers on the overall purpose of the survey, the intent of specific questions and how to manage both asking questions and recording answers. In the case of both phone and face-to-face surveys, interviewers participate in mock interviews in order to gain familiarity with the questionnaire. These training sessions can highlight ways to improve the administration of the survey so that questions are clearly communicated and answers correctly recorded.

Once fieldwork begins, interviews are regularly monitored by supervisors. For phone surveys, this typically involves a supervisor listening in to a live interview or calling back a respondent to verify that an interview was completed with an eligible individual. Similarly, with respect to face-to-face surveys, supervisors will travel with interview teams to urban neighborhoods or rural villages to make certain that interviewers visit residences randomly drawn from a list or selected randomly from a pre-determined route. Supervisors will also later visit a certain percentage of residences to confirm that eligible individuals have been interviewed.

Quality checks during the survey administration process help ensure that Pew Global Attitudes surveys reach their target populations and ask the intended questions. By using careful field supervision, appropriate sample design and thorough translations, we can be sure that our surveys accurately represent public opinion around the world.

Presidential Address delivered by Scott Keeter
Director of Survey Research, Pew Research Center
at the
67th Annual Conference of the American Association for Public Opinion Research
Orlando, Florida
May 18, 2012

(The views expressed in this speech are those of the author).

I am honored to have this opportunity to address the 67th annual conference of AAPOR, an organization I have loved since I first joined in 1986.

For those of us in the business of studying the attitudes, behaviors and experiences of the public, these are the best of times and the worst of times.

Never before in history has so much information about the public been so readily available for us to study and analyze. The world is getting flatter, traditional authorities are losing power, and people are gaining the ability to organize and act without a hierarchy to propel and guide them. In this changing world, information about what people do, what they think and what they want is more critical than ever.

But the institutions dedicated to measuring what the public thinks, experiences and does are undergoing significant change – and much of this change is not good. At the same time, our trusted methods for gathering information are encountering serious challenges, as we all know too well.

Increasing political polarization also poses a threat, as information increasingly comes to be seen through the red and blue filters of our partisan world. Perceptions of the condition of the economy by Republicans and Democrats did not differ during the 1990s, but began to diverge in the next decade and are now quite wide. Republicans rated the economy better than Democrats during George W. Bush’s time in office, and the pattern has reversed since Barack Obama became president.

And more generally, disagreeable or inconvenient information is heavily discounted by many people. When the information comes from survey research, the producers of these surveys face increasingly hostile attacks.¹

This is not a new phenomenon, but it’s arguably worse today than it was many years ago, facilitated by greater consumer choice in information sources, partisan news media, and a large and sophisticated infrastructure created to generate and disseminate information helpful to the ideological and financial interests of those who pay for it.

None of this is news to you. Among many other voices, AAPOR presidents over the past couple of decades or more have been talking about these issues in one form or another. I would like to offer a few ideas about how we as survey researchers and as AAPOR can respond.

I also want to remind us of a principle that is at the core of AAPOR, the shared point of view that unites a fairly disparate group of academics, policy practitioners, methodologists, political pollsters and others.

A unifying and foundational principle of our profession, and of AAPOR, is that the generation of unbiased information and data about the population is critical to the health of our democracy. Democracy has many meanings, but common to them is the connection between the people and those who hold power in the society. Though often imperfect in conception and implementation, democracy implies that people are equal. And so, biased information about the public weakens the connection at the core of democracy.

Indeed, this core commitment to unbiased data, I would argue, is one of the main reasons why AAPOR continues to attract such a diverse group of researchers: public opinion pollsters, health researchers, sociologists and demographers, and those who focus mainly on the craft and science of drawing samples, designing questionnaires and collecting data.

What are the threats?

When we think about the threats to our profession, many of us probably think first about growing non-response, non-coverage and other methodological challenges we work constantly to overcome. These are formidable. We probably used the term formidable years ago when response rates headed well below the norms we were trained to expect. Depending on the sector in which you worked, that meant below 50 percent, or 40 percent, or 30 percent, or lower. Today, many of us in the public opinion sector are in the single digits using conservative rules for estimating the rates.

On the matter of how well we cover the population of interest, even in the fall of 2004, I got dozens of questions about the potential of bias from cell phone-only households in presidential election polls. Cell-only households were 7 percent of all households then. They are probably five times that now. We know there is a potential for bias now.

But tough as these are, they are pretty familiar, so I’d like to focus on some other threats that are equally dangerous, and less amenable to the scientific solutions we have tried to apply to the growing wireless-only population and high refusal rates.

For some of us old timers in AAPOR, the very technologies and methods talked about in this conference constitute a threat to our way of life. Whether we are talking about opt-in internet panels, which have been around for a while, or the non-survey methods such as automated content coding of social media, the integration of data from what has been called the “internet of things,” and from so-called big data more generally, these have drawn interest and resources away from traditional surveys. I will come back to this shortly.

Another is the financial struggle of American journalism, the institution that synthesizes much of the information we produce and provides it to policymakers and the public. News organizations are also producers of survey research or clients of organizations that do. As my colleagues in the Project for Excellence in Journalism have documented, journalism does not have an audience problem – it has a money problem. Even as the audience for mainstream news organizations has remained stable or even grown, revenues have plummeted.

The digital revolution is responsible for much of the financial problem and may be part of the solution, but to date the balance has been heavily weighted to the problem side. For example, regarding newspapers in 2011, my Pew colleagues reported that “… losses in print advertising dollars outpaced gains in digital revenue by a factor of roughly 10 to 1, a ratio even worse than in 2010. When circulation and advertising revenue are combined, the newspaper industry has shrunk 43% since 2000.” The same may be true in other news media, or will be soon.

Diminished revenue has led to staffing reductions and cuts in newsgathering operations, including polling. News organizations have reduced their polling budgets, cut staff or even eliminated their polling operations. Not only does this reduce the volume of new survey research being conducted, but it also deprives the remaining news staff of expertise about quantitative data that can help them make sense of surveys conducted by others and make informed judgments about the difference between good data and bad data.

Indeed, the market for good data has arguably eroded. There is evidence that many people don’t or can’t make distinctions between the careful fact-checked reporting of major news organizations and the on-the-fly postings of blogs and opinion sites masquerading as news sites. The competitive pressures in this environment may even lead to an erosion of standards in major media, just to keep up.

Another problem we confront is that financial pressures in higher education have led to the closing of some university-based survey research centers, and significant challenges to those that remain in business. By all accounts, most of the centers are responding well to these challenges, but not all are. As higher education has had to retrench due to declining state support and other pressures, centers have increasingly become dependent on external funding.

The competitive environment among survey research contractors is very intense right now, as those of you in both the private and public sectors know well. But the academic centers have played a special role in our profession, providing leadership on methodological experimentation and educating the next generation of AAPOR members. Anything that threatens their health should be a concern for all of us.

Perhaps paramount among the challenges are those faced by the federal statistical system, including the U.S. Census Bureau. All of us in the survey business, regardless of our political orientation, our sector or the methods we use depend upon the federal statistical system for a data infrastructure of rock-solid national parameters.

Historically, most of the system has been safe from political pressures. There certainly has been political pressure on the use of data from the system, but the production of the data was mostly insulated from such pressure, with the notable exception of the controversy over census undercounts and how they might be adjusted.^₂

This has changed recently. A little more than a decade ago, politicians began voicing complaints about what they described as the “intrusiveness” of questions in federal surveys, particularly on what was then called the long form of the census. Even George W. Bush, when he was a candidate for president, said he wasn’t sure if he’d answer the long form if he got it.

Then last week, news arrived that the House of Representatives had voted to eliminate the American Community Survey and the Economic Census, after threatening merely to cut the budget and make compliance voluntary. Sponsors of this legislation charge that the ACS is unconstitutional.

Even if the ACS is not killed, deep cuts in the bureau’s budget loom, along with the elimination of the mandatory requirement for the ACS. If these happen, data quality will suffer. Budget cuts also are threatening other projects at Census, and in the Bureau of Labor Statistics and the Bureau of Economic Analysis.

AAPOR and all of the chapters have signed a letter urging Congress to reverse this action when final appropriations legislation is considered, but the threat is grave, even if the ACS is not, in the end, completely eliminated. Everyone in this room knows how important the census data are for our work, a point reiterated last night in the plenary session on non-probability sampling. Without those national parameters, all methods – probability or non-probability alike – will have a much more difficult time judging and eliminating bias in their data.

Opportunity

Where there is threat, there is sometimes opportunity. Given the weaknesses increasingly evident in our traditional way of doing things, we have to find alternatives.

I can’t find anything good in the attacks on the federal statistical service, and the financial pressures on the institutions that have sponsored a lot of survey research don’t appear to have an upside. But the innovations that seem to compete with our approach may not be the foe they appear to be at first sight.

For one thing, they are made possible by a strange paradox. We can’t get very many people to talk with us when we reach out to them for a survey. Maybe concerns about privacy have a lot to do with this. But when they interact with their friends – defined rather loosely – on social media and in other digital places, they appear to be very generous with personal details. Perhaps some of what we want to know from people is available from their interactions with friends and acquaintances that are publicly available. At least that’s what we hope.

The new world of social research relies to a great extent on what Bob Groves called “organic data” in his essay on the three eras of survey research in Public Opinion Quarterly 75th anniversary issue.³ By organic data he means data available from systems such as the internet and social media. From the perspective of the survey researcher, the new world of organic data is just in its infancy. We don’t fully understand what can be extracted from it. I don’t think it can be understood yet.

We do know, however, that never before has so much expression of public opinion — in venues such as Facebook and Twitter, on blogs, digital petitions — been so accessible to researchers for analysis. Never before has so much information about people’s behaviors, everything from their shopping, commuting, traveling, to their internet searches and reading, been available for linking to survey data and to their expression of sentiment in social media.

For many, this is an Orwellian nightmare, where Big Brother is replaced or joined by businesses to keep a constant eye on us. And as Kenneth Prewitt, a former census director, has stressed, the quality of data in databases and in administrative records is questionable.⁴ At a minimum, it has not been subjected to the kinds of scrutiny and study that characterizes the world of survey research.

But it is undeniably a completely new age for the research world. We have gone from the invention of the probability sample to “big data” in the lifetime of some of the people in this room.

This conference has as its theme the opportunities available to us in this new frontier. A year ago in Phoenix, I met Michael Schober, who along with Fred Conrad wrote Envisioning the Survey Interview of the Future. He pitched the idea of inviting scholars doing cutting edge work outside of our paradigm to come to this year’s conference to address these issues. The stars aligned, because many people, notably conference chair Dan Merkle, were also thinking along these lines. Dan had no trouble selling the AAPOR Executive Council on this conference theme.

I think this is terrific. Even my organization, the Pew Research Center, which has been quite conservative on the methodological front, has been doing and reporting on such methods as computer-assisted coding of blogs, tweets and news organizations’ websites.

Last year, in his presidential address Frank Newport said we should “promote flexibility, not dogmatism, in which methods to use….”⁵ I’ll see him on “flexibility” and raise him to “enthusiastic experimentation.” My non-scientific conversations with many of you lead me to believe that much of our membership is eager to find out what can be learned with these new approaches. But – as is also typical of the AAPOR approach – we want to judge the quality and applicability of these methods carefully before jumping into the pool. After all, the conference theme is Evaluating New Frontiers.

Three things we can do to keep survey research relevant for democracy

As we think about the threats to survey research and the opportunities available to us, I would urge us to do three things to help keep survey research relevant for democracy.

1. Remember why random samples are important

The first thing I will say will seem obvious to some; it will sound contradictory to others, after my embrace of the new frontier. That is to remind us of the importance of the paradigm that has been the foundation of the work of most of us in the association for most of the time we have been doing it – and that is the production and analysis of what Groves labels “designed data.” We have depended upon it for 75 years.

The linchpin of designed data was the insight of the founders of modern statistics and survey research – the idea that every object in the population would have a known chance of being included, and thus, at least in the final analysis, an equal chance of being included. In surveys, this is highly desirable. In a democracy, this is absolutely essential.

I cannot put this more eloquently than it was said by Sidney Verba, the eminent political scientist who wrote the following passage in his presidential address to the American Political Science Association in 1995:

Verba said: “Surveys produce just what democracy is supposed to produce – equal representation of all citizens. The sample survey is rigorously egalitarian; it is designed so that each citizen has an equal chance to participate and an equal voice when participating.”⁶

The paradigm of most of the survey work of AAPOR’s members is the probability sample, to use Verba’s words, the “rigorously egalitarian” method that helps to remove, or at least reduce, the biases associated with literacy, education, wealth and other factors that make the voices of some people louder and clearer than others. Verba was making this point to scholars of political behavior, but it applies to all of us in the survey world.

If we edge away from the probability model as we explore the new frontier, we must keep an eye on how well we are representing populations that aren’t as present on the internet, Facebook, Twitter or the commercial and credit databases that can be mined for insights.

In research that my colleagues at the Pew Internet and American Life Project did with Verba and his colleagues, online forms of participation were found to increase the percentage of young people engaged in certain political acts. But the broader takeaway was that the same biases we see in traditional outlets of participation – voting, working for a campaign, communicating with public officials – are still present online. Even though more young people are doing these things, the better educated and more affluent are overrepresented among the activists. As a consequence, substituting analysis of online political participation for survey-based measures will come with a bias toward the better educated and more affluent.

And so I am hoping that we remain mindful about the biases inherent in our new methods. We must not let our tools dictate what we study. We must shape our tools to answer our questions. Bob Groves put it very well, as he usually does, in his Public Opinion Quarterly essay. He says: “The challenge to the survey profession is to discover how to combine designed data with organic data, to produce resources with the most efficient information-to-data ratio.” To which I’d add: “And that don’t privilege the voices and experiences of the wired, the articulate and the highly motivated.”

2. Defend yourselves (and others) in the industry

The second thing I’m recommending is that we all do everything we can to defend high quality survey research, its producers and those who distribute it. We are all accustomed to criticism from those who don’t like our findings, but in the 24-7 speeded-up news cycle, with highly partisan bloggers and news organizations playing a bigger role today, this type of criticism is more prevalent than ever.

Information not only informs policy making, but serves as a political weapon. Perhaps it always has, but I have a sense that bad information, whether it’s junk science, economics or polling data, is now more widespread.

And then there are the familiar attacks on polling that those of us in the opinion research world have been dealing with for decades – that polls are used by politicians to manipulate the public; that horse race journalism is a disease brought on by horse race polling; that public opinion polls turn leaders into followers; that polls create opinions where none exist. All of these have a bit of truth in them, and all have been around since near the beginning of modern polling.

Andrew Kohut, president of the Pew Research Center, spoke to the issue of defending polling in his presidential address to this conference in 1995.⁷ He looked at the criticisms leveled against polling 50 years prior, and found at least two notable things. First, the criticisms were very familiar. Second, the response of the pioneers of the survey industry – George Gallup, Harry Field, Paul Lazarsfeld and others – was swift and sharp. But he observed that the pollsters of 1995 were not so quick to defend polling and its role in democracy.

At AAPOR we are fighting a mostly defensive war. We try to address allegations of push-polling and other instances of groups using surveys as a cover for what they are really trying to do. We sign letters and protest attacks on survey research generally and on the federal statistical service.

One of the things we are doing pro-actively to address specific criticisms about the role of polls in our democracy is a task force headed by Bob Shapiro and Frank Newport, the “Public Opinion and Leadership Task Force.” Among other things, this task force is dealing with concerns about how credible is public opinion on issues where public knowledge is low and preferences are weak. The task force aims to provide guidance on when polls provide a reliable indicator of public sentiment and when they should be accorded less confidence.

And, as I noted earlier, the federal statistical service is critical to us. The song says you don’t know what you’ve got till it’s gone, but WE know. We must defend its funding and its political independence.

3. Promote transparency in the use of the new methods… and the old ones

My last recommendation is short and familiar: to promote and practice full transparency in the use of the methods on the new frontier – and in the use of the old methods. A cursory review of reporting from most online opt-in panels indicates that transparency is not their strong suit, to put it mildly. For non-survey methods, the record may even be worse. But even in the world of traditional survey methods, we’ve learned from the Transparency Initiative that many of our members – the best intentioned citizens of the survey world – struggle with the details of documenting their methods.

In his presidential address to AAPOR in 2010, Peter Miller rightly pointed out that a lack of transparency in survey research is a grave threat.⁸ Its absence contributes to ignorance about surveys. It makes fraud more likely. It contributes to the growth of cynicism about surveys. And it allows some to make a false equivalence among sources of data, allowing good survey data to be assigned to the lowest common denominator of information.

A lack of transparency also fails to take advantage of one of the great benefits of the new information environment, which is peer review and pressure. This is the good side of the enhanced scrutiny and criticism we get. Especially for those of us polling in the political world, a sophisticated critic is likely watching what we are doing and will call us out if we make mistakes. Being transparent won’t immunize us, but at least it reduces the chance that our motives will be questioned. And if we are transparent, flaws and limitations may be quickly discovered, so that they can be corrected, and we can do better next time.

Transparency, whether in the old methods or the new ones, makes it a lot easier for us to defend the work of our colleagues. If we know what they did and how they did it, we can defend them from a position of strength.

Conclusion

So while I worry about the challenges we are facing, I am confident that AAPOR and its members are up to meeting those challenges. Our history shows that we are. AAPOR and its members led the field’s response to previous upheavals. Our conferences feature research on the new problems well before answers emerge in the research literature – just think about the cell phone coverage issue. Competitors share knowledge freely, even though doing so may lessen their advantage in the marketplace. And there remains an open-mindedness that makes it possible to have a conference theme about non-survey methods and non-probability sampling approaches, and have a big audience for a plenary featuring a respectful debate on the subject. That’s the AAPOR I know and love.

Thank you for your attention, and I look forward to many more years in AAPOR with all of you.

[1] Examples include Republican protests against the pollster of the

Minneapolis Star-Tribune during the 2004 presidential election, and attacks on the Gallup Poll by MoveOn.org in the same

year.
[2] Kenneth

Prewitt. 2010. The U.S. Decennial Census: Politics and Political Science. Annual Review of Political Science, Vol. 13,

pp. 237-254, 2010.
[3] Robert M. Groves. “Three Eras of Survey Research.” Public Opinion Quarterly 75 (Special

Issue 2011): 861-871.
[4] Stephen

E. Fienberg and Kenneth Prewitt. “Save Your Census.” Nature 466 (26 August 2010): 1043.
[5] Frank

Newport. “Taking AAPOR’s Mission to Heart.” Public

Opinion Quarterly 75 (2010): 593-604.
[6] Sidney

Verba. “The Citizen as Survey Respondent: Sample Surveys and American

Democracy.” American Political Science

Review 90 (March 1996): 1-7.
[7] Andrew Kohut. “Opinion Polls and the Democratic

Process, 1945-1995.” Public Opinion

Quarterly 59 (1995): 463-471.
[8]  Peter V. Miller.

“The Road to Transparency in Survey Research.” Public Opinion Quarterly 74 (2010): 602-606.

The Pew Research Center often receives questions from visitors to our site and users of our studies about our findings and how the research behind them is carried out. In this feature, senior research staff answers questions relating to the areas covered by our seven projects ranging from polling techniques and findings, to media, technology, religious, demographic and global attitudes trends. We can’t promise to respond to all the questions that we receive from you, our readers, but we will try to provide answers to the most frequently received inquiries as well as to those that raise issues of particular interest.

If you have a question related to our work, please send it to info@pewresearch.org.

• How do those who don’t search for health information online differ from those who do?
• Isn’t it difficult to poll in some countries? And how confident are you in the poll findings?
• See other recent Ask the Expert questions

Q: You’ve reported on how many Americans have turned to the Internet for health care information. What about those who have not taken part in this trend? How do they differ from those who go online for this information?

A: Education significantly affects someone’s likelihood to have internet access, which of course influences a person’s likelihood to search for health information online. For example, when looking at adults who have less than a high school education, just 42% go online and, of those, 62% say they gather health information online. That means three-quarters of U.S. adults who have less than a high school education say they do not get health information online. By comparison, college graduates are nearly all online (94%) and 89% of that group gathers health information online. Therefore, only about one-in-six U.S. adults with a college degree say they do not get health information online.

In 2002, we asked a basic screening question to see if respondents ever use the internet to look for health information or medical advice. Of those who answered in the negative, nearly half of internet (47%) users said the major reason they did not search for health information online was that there were not any health or medical issues of immediate concern to them. Almost the same number (46%) said they were satisfied with the health and medical information they got elsewhere. A smaller number (12%) said that much of the information on the Internet could not be trusted and 9% said they would not know where to start looking for health information online.

Susannah Fox, associate director, digital strategy, Pew Internet & American Life Project

Q. I’ve never been called for a poll. What are the odds I’d be selected for a survey? And, if my home is called, how do you determine who you talk to?

You have roughly the same chance of being called as anyone else living in the United States who has a telephone. This chance, however, is only about 1 in 154,000 for a typical survey by the Pew Research Center for the People & the Press. To obtain that rough estimate, we divide the current adult population of the U.S. (about 235 million) by the typical sample size of our polls (usually around 1,500 people). Telephone numbers for Pew Research Center polls are generated through a process that attempts to give every household in the population a known chance of being included. In practice, that’s difficult to achieve. For one thing, we avoid calling large blocks of telephone numbers that have not been assigned to any households. Because the process of identifying such blocks of numbers is imprecise, some households may get excluded from the sample. Similarly, you will have a greater chance of being called if you have both a cell phone and a landline phone (since you have two different ways of landing in our sample). And if you don’t have a telephone at all (about 2% of households), then of course you have no chance of being included in our telephone surveys.

Once numbers are selected through random digit dialing, the process of selecting respondents within a household is different for landline and cell phone numbers. When interviewers reach someone on a landline phone, they randomly ask half the sample if they can speak with “the youngest male, 18 years of age or older, who is now at home” and the other half of the sample to speak with “the youngest female, 18 years of age or older, who is now at home.” If there is no eligible person of the requested gender at home, interviewers ask to speak with the youngest adult of the opposite gender, who is now at home. This method of selecting respondents within each household helps us obtain more interviews with young people, who are often more difficult to reach than older people because of their lifestyles. One implication of selecting a respondent within a household is that the probability of being interviewed for the survey depends on how many adults live in the household and are at home at the time of the call.

Unlike a landline phone, a cell phone is assumed to be a personal device. As a result, for cell phone calls we do not attempt to select a respondent from all of the adults in the household. Instead, we ask if the person who answers the cell phone is 18 years of age or older. If they are 18 or older, we attempt to interview them.
Once we’ve completed a survey, we adjust the data to correct for the fact that some individuals (e.g., those with both a cell phone and a landline) have a greater chance of being included than others. For more on how that’s done, see the discussion of weighting in our detailed methodology statement.

Scott Keeter, Director of Survey Research, Pew Research Center
President, American Association for Public Opinion Research, 2011-2012

Q. Each year, the Pew Research Center’s Global Attitudes Project conducts public opinion polls around the world. Isn’t it difficult to poll in some countries? And how confident are you in the poll findings?

There is no question that the process of fielding surveys in foreign countries can be challenging and even difficult, especially as the majority of our polls are conducted through face-to-face interviews. That said, the rigorous quality standards we apply to our Pew Global Attitudes survey work abroad are the same as those we apply here in the U.S. Namely, we strive to ensure that our surveys accurately represent the populations of individual countries, and that we ask questions that deliver valid information about how people view important events and issues of the day.

The task of ensuring our surveys represent the adult-age population a foreign country requires employing rigorous sampling methods.  The first step is to work closely with our principal research partners to identify local polling firms with knowledgeable staff and proven track-records in designing large survey samples. The good news is that political and economic changes over the past two decades have generally increased the demand for social and market research and have made it easier for us to find local, capable research firms.

The not-so-good news is that, compared with the U.S., the penetration of landline phones in many countries has not gotten to a point at which we have felt comfortable fielding national surveys by phone. And while mobile phones are quickly becoming an integral part of life for many people around the globe, reliable information about mobile users and how to integrate them into national samples remains limited. Thus, with the exception of a few countries such as Britain, France, Germany and Japan, we have erred on the side of reliability and have administered our surveys in person, rather than by phone. This means that unlike national surveys in the U.S., which can be completed in just days, most Global Attitudes surveys take two or more weeks to complete.

Due to their use of proven sampling techniques, the local vendors we work with can achieve nationally representative surveys by conducting face-to-face surveys with about 1,000 respondents. The key is ensuring that these respondents are selected in a random, unbiased fashion – and that all adult members of a country’s population are eligible for inclusion in a survey. More often than not, we meet these requirements by using multi-stage, cluster samples.

What this means is that rather than randomly selecting individuals directly (by phone, for example), we first randomly select clusters of individuals – beginning with relatively large territorial units, akin to counties in the U.S. Once these primary clusters are selected, we randomly select smaller territorial units, until we work our way down to city blocks or villages. At this stage, interviewers either visit addresses selected randomly from a list, or they follow a so-called “random walk” in which they visit every third or fourth residence along a set route. At each residence, interviewers randomly select a respondent by using a Kish grid (a detailed list of all household members) or by selecting the adult who has had the most recent birthday.

As mentioned above, in most countries surveyed by Pew Global Attitudes, multi-stage, cluster samples are used to field nationally representative surveys. However, in a few instances we are unable to conduct full, national surveys face-to-face. Sometimes, the limiting factors are cost and time. In China, for example, it would take many weeks to collect face-to-face interviews from across the country, and it would be prohibitively expensive to transport trained interviewers long distances. Therefore, for now our surveys represent only 57% of the Chinese population (those mostly in urban areas). We hope to expand our coverage in the future.

In other instances, concern for the safety of interviewers keeps us from fielding truly national surveys. In Pakistan, for example, our surveys exclude 15% of the population due to concerns about sending interviewers into frequently violent border regions. As in China, our survey respondent base is predominantly urban because of this. But regardless of the scope, our surveys in China and Pakistan are held to the same methodological standards as our surveys conducted elsewhere.

One of the quality checks we perform for all Pew Global Attitudes surveys is to compare the demographic characteristics of the people included in our surveys with census or other official data that describe the gender, age or educational makeup of the population. In cases where our data departs more than a few percentage points from official statistics, we may decide to adjust our data through the mathematical procedure of weighting. Weighting is a common practice in survey research. Properly applied, it can improve our ability to accurately report how prevalent or variable attitudes are in a given society.

For all our surveys, of course, we calculate country-specific margins of sampling error, based on the number of people surveyed and whether the survey was based on random dialing by phone or a multi-stage, cluster sample. These margins of error are integral to our ability to identify attitudinal shifts or statistically-significant differences. Over the years, our key trends have proven highly reliable; they have moved in directions that track well with political and economic developments or have remained relatively stable in the absence of major events or changes at the local, regional or global level. This is another reason we stand behind the accuracy of our data.

Beyond the technical aspects of sample design, our confidence in Pew Global Attitudes’ findings stems from the time and effort we put into the design and translation of our questionnaires. In a given year, our survey questions may be translated into twenty or more languages. Coordinating so many translations can be daunting. Fortunately, since the Pew Global Attitudes Project began in 2002, we have assiduously archived translated versions of the annual questionnaire. As we develop each new round of the survey, we are able to draw on these tried-and-tested translations when we repeat specific items or trends.

But past translations are of little help when it comes to new questions or new countries. In such instances, we rely on local polling firms to first translate our questions into the appropriate local language or languages. As a standard operating procedure, we then have the translation re-translated into English by a bilingual person who has not seen the original English-language questionnaire. This is called back-translation, and it is highly useful for identifying questions that have been poorly or incorrectly translated. For complex questions or especially challenging languages, we often take the additional step of consulting with linguistic experts to perfect translation of new items.

Regardless of language, the goal of the translation process is always the same: to ensure that we ask questions that reflect our intended meaning, with results that can be compared cross-nationally.

A final reason for confidence in our findings is the careful training of interviewers and close supervision of fieldwork. In each country, prior to fieldwork, local research firms train their interviewers to properly administer the questionnaire. This includes briefing interviewers on the overall purpose of the survey, the intent of specific questions and how to manage both asking questions and recording answers. In the case of both phone and face-to-face surveys, interviewers participate in mock interviews in order to gain familiarity with the questionnaire. These training sessions can highlight ways to improve the administration of the survey so that questions are clearly communicated and answers correctly recorded.

Once fieldwork begins, interviews are regularly monitored by supervisors. For phone surveys, this typically involves a supervisor listening in to a live interview or calling back a respondent to verify that an interview was completed with an eligible individual. Similarly, with respect to face-to-face surveys, supervisors will travel with interview teams to urban neighborhoods or rural villages to make certain that interviewers visit residences randomly drawn from a list or selected randomly from a pre-determined route. Supervisors will also later visit a certain percentage of residences to confirm that eligible individuals have been interviewed.

Quality checks during the survey administration process help ensure that Pew Global Attitudes surveys reach their target populations and ask the intended questions. By using careful field supervision, appropriate sample design and thorough translations, we can be sure that our surveys accurately represent public opinion around the world.

Jim Bell is Director of International Survey Research for the Pew Research Center.

You can find a list of the surveys by the project here.

For more “Ask the Expert” questions, click here.

The presidential campaign is moving into high gear and many polling organizations are, in fact, conducting surveys only of registered voters, or even in some cases only likely voters. This is not the approach of the Pew Research Center, which studies public attitudes toward politics, the press and policy issues. On subjects ranging from the debt crisis to the war in Afghanistan — issues that affect all Americans, voters and non-voters alike — the Center attempts to get the broadest possible measure of public attitudes.

This is not to say that the Center does not track the electoral preferences of voters. While we ask election questions — indeed, all questions — of the public, we present the results of election questions based on registered voters in our survey reports. Registered voters are adults who say that they are “absolutely certain” they are registered to vote in their precinct or election district (usually around three-quarters of all survey respondents). Typically, the views of registered voters are not very different from those of the general public. Yet on election questions, we feel it is important to present the results based on those who are at least certain they are registered to vote; a person who tells us they are not registered, or is not sure, is very unlikely to cast a ballot. As the election approaches in fall 2012, we will increasingly report on the preferences and attitudes of likely voters as well as registered voters. More detail on how we identify likely voters is available here.

In our July 28 survey report — see “Obama Loses Ground in 2012 Reelection Bid” — 41% of registered voters said they would like to see Obama reelected while 40% said they would prefer that a Republican candidate win the election. The accompanying topline questionnaire shows the responses of all adults — as well as registered voters — on this question: 42% of all adults said they favored Obama’s reelection while 37% preferred a Republican. The slight Obama edge among all adults is not unusual; those who tell us they are not registered to vote include more young adults and minorities — groups that tend to be more supportive of Obama.

On election questions, the Pew Research Center wants to provide an accurate gauge of voter intentions. Yet the Center also has conducted extensive research on non-voters — who they are, what they think, and why they do not vote. For more, see “The Party of Non-Voters” from October 2010 and “Who Votes, Who Doesn’t, and Why” from October 2006.

Carroll Doherty, Associate Director, Pew Research Center for the People & the Press