In 2020, Pew Research Center launched a new project called the National Public Opinion Reference Survey (NPORS). NPORS is an annual, cross-sectional survey of U.S. adults. Respondents can answer either by paper or online, and they are selected using address-based sampling from the United States Postal Service’s computerized delivery sequence file. The response rate to the inaugural NPORS was 29%, and subsequent years’ surveys are being designed with the same rigorous approach.
NPORS estimates are separate from the American Trends Panel (ATP) – the Center’s national online survey platform. Pew Research Center launched NPORS to address a limitation that researchers observed in the ATP. While the ATP was well-suited for the vast majority of the Center’s U.S. survey work, estimates for a few outcomes were not in line with other high-quality surveys, even after weighting to demographics like age, education, race/ethnicity and gender. For example, in 2018, roughly one-quarter of U.S. adults were religiously unaffiliated (e.g., atheist, agnostic or nothing in particular), according to the General Social Survey and the Center’s own telephone-based polling. The ATP, however, estimated the religiously unaffiliated rate at about 32%. The Center did not feel comfortable publishing that ATP estimate because there was too much evidence that the rate was too high, likely because the types of people willing to participate in an online panel skew less religious than the population as a whole. Similarly, the ATP estimate for the share of U.S. adults identifying as a Democrat or leaning to the Democratic Party was somewhat higher than the rate indicated by the General Social Survey and the Center’s own telephone-based polling.
From 2014 to late 2020, the Center approached these outcomes slightly differently. We addressed the political partisanship issue by weighting every ATP survey to an external benchmark for the share of Americans identifying as a Republican, Democrat or independent. For the benchmark, we used the average of the results from our three most recent national cellphone and landline random-digit-dial (RDD) surveys. During this time period, ATP surveys were not, however, weighted to an external benchmark for Americans’ religious affiliation. The ATP was used for some research on religious beliefs and behaviors, but it was not used to estimate the overall share of Americans identifying as religiously affiliated or unaffiliated, nor was it used to estimate the size of particular faith groups, such as Mormons, Catholics or Protestants. NPORS allows us to improve and harmonize our approach to both these outcomes (Americans’ political and religious affiliations).
Design and estimates
Click here to see the latest NPORS estimates as well as methodological details. Data collection for NPORS is performed by Ipsos.
Why is the NPORS response rate higher than most opinion polls?
Several features of NPORS set it apart from a typical public opinion poll.
- People can respond offline or online. NPORS offers two different ways to respond: by paper (through the mail) or online. The paper option brings in more conservative, more religious adults who are less inclined to take surveys online.
- Monetary incentives. When sampled adults are first asked to respond to NPORS online, the mailing contains a $2 pre-incentive payment and offers a $10 post-incentive payment. When nonrespondents to that first stage are sent the paper version of the survey, the mailing contains a $5 bill. These incentives give people a reason to respond, even if they might not be interested in the questions or inclined to taking surveys in general.
- Priority mailing. The paper version of the survey is mailed in a USPS Priority Mail envelope, which is more expensive than a normal envelope, signaling that the contents are important and that the mailing is not haphazard. It helps people distinguish the survey from junk mail, increasing the likelihood that they open and read what is inside.
- Low burden. The NPORS questionnaire is intentionally kept short. It’s about 40 questions long, including demographics such as age, gender and education. This means that NPORS takes about seven minutes to finish, while many polls take 10 minutes or longer.
- Bilingual materials. In parts of the country with sizable shares of Hispanic Americans, the materials are sent in both English and Spanish.
- No requirement to join a panel. NPORS respondents are not required to join a survey panel, which for some people would be a reason to decline the request.
These features are not possible in most public polls for a host of reasons. But NPORS is designed to produce estimates of high enough quality that they can be used as weighting benchmarks for other polls, and so these features are critical.
Why a “reference” survey for public opinion?
The “R” in NPORS stands for “reference.” In this context, the term comes from studies in which researchers calibrate a small sample survey to a large, high-quality survey with greater precision and accuracy. Examples of reference surveys used by researchers include the American Community Survey (ACS) and the Current Population Survey (CPS). While the NPORS is not on the scale of the ACS or CPS, nor does it feature face-to-face data collection, it does have something that those studies lack: timely estimates of key public opinion outcomes. Other studies like the American National Election Survey (ANES) and the General Social Survey (GSS) collect key public opinion measures, but their data is released months, if not years, after data collection. The ANES, while invaluable to academic researchers, also excludes non-citizens which constitute about 7% of adults living in the U.S. and are included in the Center’s surveys.
NPORS is truly a reference survey for Pew Research Center because researchers weight each ATP wave to several NPORS estimates. In other words, ATP surveys refer to the NPORS in order to represent groups like Republicans, Democrats, religiously affiliated adults and religiously unaffiliated adults proportional to their share of the U.S. population. The ATP weighting protocol also calibrates to other benchmarks, such as ACS demographic figures and CPS benchmarks for voter registration status and volunteerism.
Pew Research is weighting on political party affiliation, but isn’t that an attitude?
It’s correct that whether someone considers themselves a Republican or a Democrat is an attitude, not a fixed characteristic, such as year of birth. But there is a way to weight on political party affiliation even though it is an attitude and without forcing the poll’s partisan distribution to align with a benchmark.
Pew Research Center started implementing this approach in 2021. It begins with measuring the survey panelists’ political party affiliation at a certain point in time (e.g., every August). Ideally, the reference survey will measure the same construct at the same point in time. We launched NPORS because we control its timing as well as the ATP timing, allowing us to achieve this syncing. NPORS fields from June to August, basically overlapping with the measurement on the ATP. From September through August of the following year, we might conduct roughly 25 surveys on the ATP. For each of those surveys, we append the panelists’ party affiliation answers from August to the current survey. For illustration purposes, let’s say that a survey was conducted in December. When researchers weight the December ATP survey, they take the August answers and weight them to the NPORS estimates for the partisan distribution of U.S. adults during the June to August timeframe. If, for example, Democrats were more likely than Republicans to respond to the December survey, the weighting to the NPORS target would help reduce the differential partisan nonresponse bias.
Critically, if the hypothetical December poll featured a fresh measurement of political party affiliation (typically asked about three times a year on the ATP), the new December answers do not get forced to any target. The new partisan distribution is allowed to vary. In this way, we can both address the threat from differential partisan nonresponse and measure an attitude that changes over time (without dictating the outcome). Each summer, the process starts anew by measuring political party on the ATP at basically the same time as the NPORS data collection.
Is the NPORS design connected to the ATP?
Respondents to NPORS, whether they complete the survey on paper or online, are not required to join the ATP. That said, there is a link in the sample design for these projects. After the vendor draws the national sample of addresses, they send a mailing that contains $2 and asks the adult with the next birthday to go online and complete a survey. The online survey is the NPORS questionnaire (about 40 questions), plus a few additional questions. At the end of the online survey, respondents are asked if they are willing to join the ATP. Their responses to the NPORS questions are used for the NPORS estimates regardless of whether they agree to join the ATP. Weeks later, addresses that did not complete the online survey are sent a Priority Mail envelope with a paper version of the NPORS questionnaire. This paper survey does not contain any additional questions, nor does it ask the respondent to join the ATP. For NPORS, researchers combine the answers from both the online and paper questionnaires and apply weighting.
Can other survey researchers use NPORS?
Yes. As a nonprofit organization, we seek to make our research as useful to policymakers, survey practitioners and scholars as possible. As with the Center’s other survey work, the estimates and data are freely available.
