June 12, 2014

How Pew Research conducted the polarization survey and launched a new research panel

Scott Keeter, Director of Survey Research, Pew Research Center

Throughout its history, the Pew Research Center has periodically conducted major surveys that take an in-depth look at important trends in American political attitudes and behavior. Today we released one such survey on political polarization, which is arguably the defining feature of early 21st century American politics. This is reflected not only in the public’s views about issues ranging from immigration to guns, but also in their personal lives and lifestyles. The study is the center’s largest-yet effort — a survey of more than 10,000 adults, as well as a new component called the American Trends Panel. The survey was funded in part through grants from the William and Flora Hewlett Foundation, the John D. and Catherine T. MacArthur Foundation and supported by Don C. and Jeane M. Bertsch.

Director of Survey Research Scott Keeter answered some questions about how the survey was done.

This survey includes more than 10,000 adults. Why did you decide to survey so many people for this report – isn’t a nationally representative sample usually around 1,000 people?

There are many reasons, but the most important is that having a larger number of people participating in the survey allows us to better describe the characteristics, attitudes and behaviors of smaller segments of the larger, nationwide public. For example, the larger sample allows us to interview a larger number of campaign donors, people with consistently conservative or liberal attitudes or regular primary voters. These individuals, even as smaller shares of the public, may have an outsize impact on the phenomenon of political polarization.

For example, in a typical poll of 1,000 respondents, only about 50-60 people will report that they have donated $250 or more to candidates or political groups in the past two years, far fewer than we’d need to describe who they are, what they think, and so forth. In our poll of 10,000 people, however, fully 570 respondents said they donated $250 or more. That size sample allows us to describe this group in much more detail.

Similarly, the large sample allows us to better understand politics on a more micro-level than we normally do. For example, we can describe the characteristics of residents of certain kinds of geographic areas such as places with highly competitive races for the U.S. House (about 4% of all districts) or places with high concentrations of recent immigrants.

Some of the responses came from something new called the American Trends Panel. What is that, and how does it work?

The American Trends Panel is a new endeavor for the Pew Research Center. We recruited survey respondents from that initial representative telephone survey of 10,013 into what’s called a panel – a group of people who agree to take monthly surveys with us. The American Trends Panel will yield many benefits. One is that it allows us to collect a great deal of additional information to go with the rich set of political questions already asked on the polarization survey. This will give us far more information about each panelist than we could reasonably expect to get in a single telephone interview. For example, we have a very comprehensive set of measures about Americans’ media habits from our first panel survey. Combined with our measures of ideological polarization and political engagement from the telephone survey, we can paint a much richer picture of how media consumption and polarization are related.

A second benefit is that we can follow individual panelists over time and see how their current views might predict future behavior and whether their views change. For example, are the most polarized respondents less likely to change their party affiliation, or how they view the other party, than are people who are less polarized? In the 2014 general elections, how well will our measures of polarization help us predict whether an individual will vote or not?

Finally, the fact that we survey about 90% of our panelists online opens up a wealth of possibilities for us in terms of the kinds of questions we can ask. We can show respondents photographs of political figures, logos of news organizations or charts and graphs of information, and thus gauge the impact of visual information on political attitudes.

Are the responses from the panel as representative and accurate as they are in a telephone survey?

Yes. Because the panel was drawn from a traditional probability sample, it is a nationally representative, probability sample. Consistent with this is the fact that the demographic and political profile of the 3,308 people who took our first panel survey is very similar to that of the overall telephone survey sample from which they were recruited. In addition, although most of our panel interviews are conducted online, panelists who do not have access to the internet are surveyed by telephone or mail to ensure that the panel represents all Americans, regardless of whether they have access to the internet.

Are people in the panel compensated for their participation? If so, why?

We do provide a small monetary incentive to panelists for each survey they complete. We do this in recognition of the sizeable time commitment that participation entails. Another factor is that better-educated and more affluent people may be more likely to do the surveys because they find them interesting, while the financial payment – though small – may be a bigger reason for low income people to do the surveys. Without an incentive, our sample would likely be more tilted toward the educated and affluent.

Can anyone join the panel?

Unfortunately, no. The American Trends Panel is made up only of members who were invited to join. This feature is necessary to ensure that the panel is representative of all U.S. adults, since everyone in the U.S. had a known chance of being interviewed in the original polarization telephone survey and invited to join the panel at that time. Allowing people to volunteer for the panel would lead to biases in the overall panel results because, then, certain kinds of people – perhaps those more interested in politics – might be more likely to find out about the panel and want to join.

What else do you plan to do with the panel?

We will continue to collect data that contributes to our study of political polarization, including information about the panelists’ reactions to the election campaigns in the fall and whether or not they vote in the November elections. But we also will gather measures of opinions and behaviors on other non-political subjects including technology use and religion.

Topics: Political Polarization, Research Methodology, U.S. Political Parties

  1. Photo of Sara Goo

    is Senior Digital Editor at the Pew Research Center.

Leave a Comment

Or

All comments must follow the Pew Research comment policy and will be moderated before posting.

6 Comments

  1. Barbara3 months ago

    I read the results of the religion study, assessing how Americans view people of various fait backgrounds. Why did you include only Catholics and Evangelicals in the Christian groups. I found it interesting that you asked Mainline Protestants what they thought of the other Christian groups, but didn’t ask what the others thought of them. Is it that Mainline Protestants are seen as irrelevant today? Is it that society has a lot of prejudice towards them, or maybe that you are?

    Reply
  2. Skip Voorhees5 months ago

    I like the concept as well as the format. My wife and I read the daily New York Times and support public radio and TV, so would possibly be seen as biased. We always vote, favoring the liberal persuasion. We enjoy the Pew reports as a valued and unbiased source of information. SV

    Reply
  3. Jeff5 months ago

    “Are the responses from the panel as representative and accurate as they are in a telephone survey?
    Yes.”

    They won’t be the same. This panel won’t be the same as a one-time, random sample. Given the probability of being included in a national poll, such one-time participants couldn’t have anticipated their inclusion and changed their behavior in any way in preparation to being questioned. That’s not true with an ongoing panel that knows it will be questioned monthly.

    It also assumes there’s no bias that exists in in the choice of, the wording or the order of questions that might have effect on the evolution of thinking of the American Trends Panel, beyond what it might have existed in the survey of the one-time panel. People who craft the topline scripts at Pew aren’t godlike figures devoid of their own prejudices and biases that could affect their choices among the universe of possible questions. I think a fair reading of the topline questionnaire of this survey would demonstrate some of that.

    Reply
  4. Paul Turner5 months ago

    Congratulations on your Political Polarization in the American Public study. I am curious if you have broken down your results to the state level?

    Thanks for your time

    Reply
  5. Sheila5 months ago

    Would love to know about opinions by income, education and age. We hear that money makes people conservative and that the young conservatives lean a bit left on social issues, but is that just the college Republicans?
    More please!

    Reply
  6. tobyw5 months ago

    As the government moves to the left of the Republican conception of what is proper and falls short of what the Dems consider proper, it is natural for Republicans to react more and more strongly.

    Reply