How Pew Research conducted the polarization survey and launched a new research panel
Throughout its history, the Pew Research Center has periodically conducted major surveys that take an in-depth look at important trends in American political attitudes and behavior. Today we released one such survey on political polarization, which is arguably the defining feature of early 21st century American politics. This is reflected not only in the public’s views about issues ranging from immigration to guns, but also in their personal lives and lifestyles. The study is the center’s largest-yet effort — a survey of more than 10,000 adults, as well as a new component called the American Trends Panel. The survey was funded in part through grants from the William and Flora Hewlett Foundation, the John D. and Catherine T. MacArthur Foundation and supported by Don C. and Jeane M. Bertsch.
Director of Survey Research Scott Keeter answered some questions about how the survey was done.
This survey includes more than 10,000 adults. Why did you decide to survey so many people for this report – isn’t a nationally representative sample usually around 1,000 people?
There are many reasons, but the most important is that having a larger number of people participating in the survey allows us to better describe the characteristics, attitudes and behaviors of smaller segments of the larger, nationwide public. For example, the larger sample allows us to interview a larger number of campaign donors, people with consistently conservative or liberal attitudes or regular primary voters. These individuals, even as smaller shares of the public, may have an outsize impact on the phenomenon of political polarization.
For example, in a typical poll of 1,000 respondents, only about 50-60 people will report that they have donated $250 or more to candidates or political groups in the past two years, far fewer than we’d need to describe who they are, what they think, and so forth. In our poll of 10,000 people, however, fully 570 respondents said they donated $250 or more. That size sample allows us to describe this group in much more detail.
Similarly, the large sample allows us to better understand politics on a more micro-level than we normally do. For example, we can describe the characteristics of residents of certain kinds of geographic areas such as places with highly competitive races for the U.S. House (about 4% of all districts) or places with high concentrations of recent immigrants.
Some of the responses came from something new called the American Trends Panel. What is that, and how does it work?
The American Trends Panel is a new endeavor for the Pew Research Center. We recruited survey respondents from that initial representative telephone survey of 10,013 into what’s called a panel – a group of people who agree to take monthly surveys with us. The American Trends Panel will yield many benefits. One is that it allows us to collect a great deal of additional information to go with the rich set of political questions already asked on the polarization survey. This will give us far more information about each panelist than we could reasonably expect to get in a single telephone interview. For example, we have a very comprehensive set of measures about Americans’ media habits from our first panel survey. Combined with our measures of ideological polarization and political engagement from the telephone survey, we can paint a much richer picture of how media consumption and polarization are related.
A second benefit is that we can follow individual panelists over time and see how their current views might predict future behavior and whether their views change. For example, are the most polarized respondents less likely to change their party affiliation, or how they view the other party, than are people who are less polarized? In the 2014 general elections, how well will our measures of polarization help us predict whether an individual will vote or not?
Finally, the fact that we survey about 90% of our panelists online opens up a wealth of possibilities for us in terms of the kinds of questions we can ask. We can show respondents photographs of political figures, logos of news organizations or charts and graphs of information, and thus gauge the impact of visual information on political attitudes.
Are the responses from the panel as representative and accurate as they are in a telephone survey?
Yes. Because the panel was drawn from a traditional probability sample, it is a nationally representative, probability sample. Consistent with this is the fact that the demographic and political profile of the 3,308 people who took our first panel survey is very similar to that of the overall telephone survey sample from which they were recruited. In addition, although most of our panel interviews are conducted online, panelists who do not have access to the internet are surveyed by telephone or mail to ensure that the panel represents all Americans, regardless of whether they have access to the internet.
Are people in the panel compensated for their participation? If so, why?
We do provide a small monetary incentive to panelists for each survey they complete. We do this in recognition of the sizeable time commitment that participation entails. Another factor is that better-educated and more affluent people may be more likely to do the surveys because they find them interesting, while the financial payment – though small – may be a bigger reason for low income people to do the surveys. Without an incentive, our sample would likely be more tilted toward the educated and affluent.
Can anyone join the panel?
Unfortunately, no. The American Trends Panel is made up only of members who were invited to join. This feature is necessary to ensure that the panel is representative of all U.S. adults, since everyone in the U.S. had a known chance of being interviewed in the original polarization telephone survey and invited to join the panel at that time. Allowing people to volunteer for the panel would lead to biases in the overall panel results because, then, certain kinds of people – perhaps those more interested in politics – might be more likely to find out about the panel and want to join.
What else do you plan to do with the panel?
We will continue to collect data that contributes to our study of political polarization, including information about the panelists’ reactions to the election campaigns in the fall and whether or not they vote in the November elections. But we also will gather measures of opinions and behaviors on other non-political subjects including technology use and religion.
Sara Kehaulani Goo is a former senior digital editor at Pew Research Center.