U.S. Survey Research
Frequently asked questions
Some general questions
Why am I never called to be polled?
You have roughly the same chance of being called as anyone else living in the United States who has a telephone. This chance, however, is only about 1 in 154,000 for a typical Pew Research Center survey. To obtain that rough estimate, we divide the current adult population of the U.S. (about 235 million) by the typical sample size of our polls (usually around 1,500 people). Telephone numbers for Pew Research Center polls are generated through a process that attempts to give every household in the population a known chance of being included. Of course, if you don’t have a telephone at all (about 2% of households), then you have no chance of being included in our telephone surveys.
Once we’ve completed a survey, we adjust the data to correct for the fact that some individuals (e.g., those with both a cellphone and a landline) have a greater chance of being included than others. For more on how that’s done, see the discussion of weighting in our detailed survey methodology.
Can I volunteer to be polled?
While we appreciate people who want to participate, we can’t base our polls on volunteers. A survey of volunteers is a “non-probability sample” and the results cannot be generalized to the public as a whole. The key to survey research is to have a random sample so that every type of person has an equal chance of having their views captured. Polls of volunteers would violate this principle since not everyone would have had an equal chance of being included. (See probability and non-probability sampling for more information.) And more specifically, the kinds of people who might volunteer for our polls are likely to be very different from the average American – at the least they would probably be more politically interested and engaged.
Why don’t your surveys ever reflect the opinions of people I know?
Chances are you don’t hang out with a group of friends that represents everyone in America. Your friends, coworkers and family are probably like you in many ways. If you were to have a group of friends that represents all of the country, you would have acquaintances who are black, white, Asian, rich, poor, Muslim, Catholic, from the South, Northeast, etc., or any combination of those attributes. Few of us are lucky enough to have such a diverse group of friends.
Why should I participate in surveys?
You should participate in surveys for many reasons. Polls are a way for you to express your opinions to the nation’s leaders and the country as a whole. Public officials and other leaders pay attention to the results of polls and often take them into account in their decision-making. If certain kinds of people do not participate in the surveys, then the results won’t represent the full range of opinions in the nation.
What good are polls?
Polls seek to measure public opinion and document the experiences of the public on a range of subjects. The results provide information for academics, researchers and government officials and help to inform the decision-making process for policymakers and others. Much of what the country knows about its media usage, labor and job markets, educational performance, crime victimization and social conditions is based on data collected through polls.
I’m on a “do not call” list. Doesn’t that prevent you from calling me?
No. Legitimate survey research is exempt from the Telemarketing Sales Rule, which was adopted by the Federal Trade Commission to fight fraud and protect consumers from harassment. The rule covers marketing but not opinion polling or market research that does not involve an effort to sell you something. Nonetheless, our telephone survey interviewing centers will honor any request not to be called.
Do pollsters have a code of ethics? If so, what is in the code?
The major professional organizations of survey researchers have very clear codes of ethics for their members. These codes cover the responsibilities of pollsters with respect to the treatment of respondents, their relationships with clients and their responsibilities to the public when reporting on polls.
Most of Pew Research Center’s pollsters belong to the American Association for Public Opinion Research (AAPOR) and subscribe to AAPOR’s code.
Some good examples of a pollster’s Code of Ethics include:
You can read Pew Research Center’s mission and code of ethics here.
How are your polls different from market research?
There are many similarities but the main difference is the subject matter. Market research explores opinions about products and services and measures your buying patterns, awareness of products and services or willingness to buy something. Our polls typically focus on public policy issues. We also try to measure topics like how voters are reacting to candidates in political campaigns and what issues are important to them in elections.
Do you survey Asian Americans?
Yes. Our surveys are representative of the entire adult population of the United States and accurately account for the full population’s diversity by age, gender, race and ethnicity, region, and socio-economic factors such as education levels, household income and employment status. We do not exclude anyone from our analyses based on his or her demographic characteristics. Asian-American responses (like all of the responses from our surveys) are incorporated into the general population figures we report. However, we are often not able to report separately on Asian Americans as a distinct group in a statistically sound way because of several limitations. Asian Americans make up a small percentage of the U.S. population—around 5.6% of the national population in the 2010 census—which makes it difficult to get a large enough sample of Asian Americans in a national survey with our standard sample size. Moreover, the diversity of languages spoken by recent Asian-American immigrants reduces the number able to respond to a typical survey and makes it more difficult and expensive to get a representative sample. In 2012, we conducted a comprehensive survey of Asian Americans in English and seven Asian languages; you can read more about the survey’s methodology here.
Collecting survey data
How did you get my number?
Most good telephone surveys of the general public use what is called a random digit dial (or “RDD”) sampling technique to generate the sample of phone numbers used in the survey. The goal is to ensure that your telephone has the same chance of being dialed as any other telephone in the United States. When using this type of telephone sample, pollsters do not know the names of the people who are called. For more information on our method of selecting telephone numbers, see random digit dialing.
How are people selected for your polls?
Once numbers are selected through random digit dialing, the process of selecting respondents is different for landline and cellphone numbers. When interviewers reach someone on a landline phone, they randomly ask half the sample if they could speak with “the youngest male, 18 years of age or older, who is now at home” and the other half of the sample to speak with “the youngest female, 18 years of age or older, who is now at home.” If there is no eligible person of the requested gender at home, interviewers ask to speak with the youngest adult of the opposite gender, who is now at home. This method of selecting respondents within each household improves participation among young people who are often more difficult to interview than older people because of their lifestyles. Unlike a landline phone, a cellphone is assumed in Pew Research Center polls to be a personal device. This means that, for those in the cell sample, interviewers ask if the person who answers the cellphone is 18 years of age or older to determine if the person is eligible to complete the survey.
What if I only have a cellphone – am I represented in your surveys?
All of the telephone surveys conducted by Pew Research Center now include people who only have cellphones (see our survey methodology in detail for more information). As the proportion of Americans who rely solely or mostly on cellphones has continued to grow, sampling both landline and cellphone numbers helps to ensure that Pew Research surveys represent nearly all adults. However, there are several challenges and extra costs associated with sampling cellphones and conducting cellphone surveys.
Don’t you have trouble getting people to answer your polls?
Yes. The percentage of people we interview – out of all we try to interview – has been declining over the past decade or more. There are many reasons for this. Some stems from the fact that people are busier and harder to reach at home. Some has to do with the use of technologies such as caller identification, voice mail and privacy managers. And some is a result of a growing unwillingness on the part of some people to be interviewed. We have done a great deal of research on whether declining response rates harm the accuracy of polls. Fortunately there is, as yet, little evidence that nonresponse is creating a serious issue with the validity of polls. (Also see the problem of declining response rates for more information)
What about people who don’t have any telephone service?
Unfortunately, for most of our surveys, people who do not have telephones are not included in the sampling frame. Because of this, they have no chance of being included in telephone surveys. Only about 2% of households have no telephone service. Without using in-person interviewing or a mail survey, there is no way to reach these phoneless households. Statistical weighting of our telephone samples helps to correct for the omission of households without any telephone service, but some bias undoubtedly remains for certain kinds of questions, especially for surveys focused on low-income populations. Because people in households with no telephone service are less likely than others to vote, their omission has not seriously damaged the accuracy of pre-election polls. This is an issue of continuing concern to pollsters. It is the subject of a great deal of ongoing research.
What demographic questions do you ask on your surveys?
There are certain demographic questions that we ask on every survey to weight the data, to make sure it’s representative of the general public. These include asking for each respondent’s age, education, race, Hispanic ethnicity and nativity, household size and ZIP code. For phone surveys, we also ask respondents whether they have a landline phone and/or a cellphone. Other demographic items that are not used for weighting, but that are used often for analysis, include a respondent’s marital status, religious affiliation and attendance, income, voter registration status, political party affiliation, political ideology, agreement with the Tea Party, labor union membership, whether the respondent is a parent and whether the respondent owns or rents his or her home. We ask these demographic questions in some, but not all, of our polls, depending on the topic. You can see the exact question wording of the English telephone demographics here and their Spanish translations here. Because Web and mail surveys require slightly different formatting of questions than phone surveys, the Web and mail versions in English are available here, and their Spanish translations are available here.
Are election polls accurate?
There has been a surge in interest in election polling as a result of highly competitive presidential elections in recent years. In the last several election cycles, most national telephone polls have been very accurate. The National Council on Public Polls compiles the election forecasts of the major national polls, and in both 2008 and 2012, these estimates were very good predictors of the final vote.
How do you know who is really going to vote?
One of the most difficult aspects of conducting election polls is determining whether a respondent will actually vote in the election. Different pollsters use different sets of questions to help identify likely voters. Overall, the aim of defining likely voters is not to predict whether individuals will vote, but to accurately estimate the preferences of the electorate. You can find more detail on how Pew Research Center identifies likely voters here.
So who’s ahead in the polls?
This commentary addresses how different polls on the presidential horse race can produce different results, particularly early in an election year.
How Reliable Are the Early Presidential Polls? Feb. 14, 2007
So Who’s Ahead? April 14, 2000
What is the “generic ballot” test?
The so-called “generic ballot” asks respondents for their vote preference without providing candidate names. This is frequently used when asking about congressional elections in a national survey; in this case, the generic question asked by Pew Research Center is: “If the elections for U.S. Congress were being held TODAY, would you vote for the Republican Party’s candidate or the Democratic Party’s candidate for Congress in your district?” where the order of “Republican Party’s candidate” and “Democratic Party’s candidate” are randomized. For those who are unsure, we ask a follow-up question asking which candidate they lean to.
Many survey organizations, including Pew Research Center, use such questions to help gauge the size of possible swings in party representation in Congress.
A version of the “generic ballot” is also sometimes used in the presidential context when there is an incumbent president. In this case, it can provide a snapshot of how an incumbent president would fare against an unnamed challenger from the opposing party. In this regard, the question is simultaneously posing a referendum on the incumbent and testing the popularity of the other party. The trend on this question can be informative, though it tends not to have predictive power, given that it is usually employed in the very early stages of the presidential race before a challenger is selected. Generic questions are rarely used once the field of nominees has been winnowed.
Do people lie to pollsters?
We know that not all survey questions are answered accurately, but it’s impossible to say that any given inaccurate answer necessarily involves lying. People may simply not remember their behavior accurately.
More people say they voted in a given election than voting records indicate actually cast ballots. In some instances, researchers have actually verified the voting records of people who were interviewed and found that some of them said they voted but did not. Voting is generally considered a socially desirable behavior, just like attending church or donating money to charity. Studies suggest these kinds of behaviors are overreported. Similarly, socially undesirable behaviors such as illegal drug use, certain kinds of sexual behavior or driving while intoxicated are underreported.
We take steps to minimize errors related to questions about socially desirable or undesirable activities. For example, questions about voter registration and voting usually acknowledge that not everyone takes part in elections. Pew Research Center’s voter registration question is worded this way:
“These days, many people are so busy they can’t find time to register to vote, or move around so often they don’t get a chance to re-register. Are you NOW registered to vote in your precinct or election district or haven’t you been able to register so far?”
Do people really have opinions on all of those questions?
People have opinions or attitudes on just about everything. Still, “I don’t know” is a legitimate answer, and people who are unsure, have no opinion or choose not to answer a question for whatever reason are always provided that opportunity.
Why do you typically ask presidential approval first in the survey?
The presidential approval question is a very important political indicator. It is a useful summary measure of the president’s standing with the public, and as such, can influence his power in dealing with the Congress, business leaders and foreign countries. We typically ask it first in the survey because we do not want any other questions to affect respondents’ answers to that question.
For example, if the survey first asks about the economy and then asks about presidential approval, the respondent may still be thinking about the economy when answering the latter question. While economic conditions may be important in assessing the president’s overall performance, so are many other issues. If the respondent is only thinking about the economy because we brought up the issue, his or her response about the president may be biased by what we call a context effect: In this case we would be priming the respondent to consider the economy in an assessment of the president.
Why are demographic questions asked at the end of the survey?
Demographic questions tend to be boring to respondents and also can seem inappropriate and threatening if asked before a level of trust is established in the interview. The interviewer wants to engage the respondent from the beginning of the conversation so that the respondent is interested in the survey and will continue to answer questions. If the interviewer started the survey by asking the respondent’s age or sex, the respondent might get bored and decide not to continue with the survey. In addition, if someone called you and first started asking how much money you make, your race, how many children you have, etc., you might be put off by these personal questions and decide not to continue with the survey.
You can see the exact question wording of English telephone demographics here and their Spanish translations here. Because Web and mail surveys require slightly different formatting of questions than phone surveys, the Web and mail versions in English are available here, and their Spanish translations are available here.
What’s all this rotating and randomizing going on in your questionnaires?
Rotating or randomizing means that questions or items in a list are not asked in the same order to each respondent. We know that answers to questions are sometimes affected by questions that precede them. By presenting questions in a different order to each respondent, we ensure that each question gets asked in the same context as every other question the same number of times (e.g., first, last or any position in between). This does not eliminate the potential impact of previous questions on the current question, but it does ensure that this bias is spread randomly across all of the questions or items in the list.
The same principle applies to the order of response options in a single question. For many questions, we randomize the order in which the answer choices are presented. That way, any effect that the order of the answer choices has on responses is spread randomly across the options.
How is form 1 different from form 2?
We often write two versions of a question and ask half of the survey sample one version of the question and the other half the second version. Thus, we say we have two forms of the questionnaire. Respondents are assigned randomly to receive either form 1 or form 2 so we can assume that the two groups of respondents are essentially identical. On questions where two versions are used, significant differences in the answers between form 1 and form 2 tell us that the difference is a result of the way we worded the two versions. For more information on question wording experiments we have conducted see question wording.
We also have different forms of the questionnaire so we can ask more questions than we would otherwise be able to ask. If we determine that half of the sample will include enough interviews for a reliable estimate, we will often ask some questions of only half of the sample. That allows us to include more questions on the survey without burdening any individual respondent with a longer interview.