January 26, 2018

For Weighting Online Opt-In Samples, What Matters Most?

Appendix A: Survey methodology

This study, sponsored by Pew Research Center, used online opt-in survey data collected by three commercial vendors. Each vendor provided their own sample and administered the survey themselves, based on a common questionnaire.

While no vendor makes the claim that that these are probability-based samples, they are intended to represent the noninstitutionalized U.S. adult population, though individuals who do not have internet access cannot join these panels and are as such not covered. The precise methods used to recruit the members of each panel are proprietary, and as a result, additional coverage properties are unknown.

Each vendor was provided with the same set of demographic quota targets based on estimates from the 2014 American Community Survey, as well as a measure of population density based on the 2010 decennial census, although differences capabilities and procedures meant that they were implemented slightly differently by each vendor. Only one vendor (Vendor 2) was able to quota on population density. Field dates and implementation details for each vendor are provided below:

Vendor 1 – June 15-28, 2016, n=10,606

Vendor 1 used sample quotas for sex by age, sex by education, age by education, census region and race and Hispanic ethnicity. Respondents sampled by Vendor 1 could opt to take the survey in English or in Spanish.

Vendor 2 – June 17-July 6, 2016, n=10,010

Vendor 2 used sample quotas for sex, age, education, census region, race and Hispanic ethnicity, and population density. Respondents sampled by Vendor 2 could opt to take the survey in English or in Spanish.

Vendor 3 – June 20-June 25, 2016, n=11,247

Vendor 3 used sample quotas on sex by age, sex by education, age by education, census region and race and Hispanic ethnicity. Spanish-language interviews (n=1,518) came from dedicated Spanish-language panels belonging to the vendor, while the rest belonged to English-language panels.