May 12, 2017

How can a survey of 1,000 people tell you what the whole U.S. thinks?

Here at Pew Research Center, we are often asked about how we conduct our research. We work hard to make our methodologies transparent and understandable, but we also know that survey mode effects and data weighting aren’t on everyone’s short list of water-cooler conversation topics.

That’s why we’re launching Methods 101, a new occasional video series dedicated to explaining and educating the public about the basic methods we use to conduct our survey research. We hope this effort will make survey methods more accessible, even if you’re not a statistician or pollster. We also hope it will help give our audience the confidence to be savvy consumers of all polls.

Our first video is about random sampling, a concept that undergirds all probability-based survey research. The video explains what it means and why it’s important. We hope you’ll find it useful.

Topics: Research Methods, Telephone Survey Methods, Web Survey Methods

  1. is director of survey research at Pew Research Center.

15 Comments

  1. Joshua Bales2 months ago

    can’t make good inferences from sample size of 1000, to population of 340 million, unless you took about 1000 samples of 1000 each, and even then we’d have to pull apart how you sampled

    1. Anonymous2 months ago

      Well, the next time you go to the doctor clinic and they want to do a blood test…tell them that’s not good enough…tell them to TAKE at least a liter!!

  2. Joshua Bales2 months ago

    n=1000 cannot accurately describe a population of 340 million people. Add in the fact that many people lie on polls and many other problems, and your statistics are failcity

    1. Anonymous2 months ago

      Yes you can. Many sophisticated sampling techniques (i.e. Multi-stage sampling) make sample sizes of about 1500-2000 very powerful. Statistically sample size stands in its own-not proportion sampled.

  3. Anonymous2 months ago

    KEEP THIS SERIES GOING!! Desperately needed and exceedingly useful; Kudos!

  4. Anonymous2 months ago

    I teach AP Statistics and I look forward to more videos in this series. What a great resource for teachers and students!

  5. Anonymous2 months ago

    What other videos will you be putting out? How soon? By September?

  6. Anonymous2 months ago

    Great video- I’ll be integrating it into my undergrad methods course this fall. THANKS!

  7. Packard Day2 months ago

    Frankly, I’d be much more confident simply using well constructed algorithms derived from metadata (e.g. Google Trends) than I would with any particular opinion poll or group of polls. Sorry Pew, but people routinely lie to pollsters. They just do.___Worse still, these same folks will also lie to their spouses, their closest family members, their friends, and even themselves as well. Go figure.

    1. Lesley Gibbs2 months ago

      As a rule people do tell the truth. You may feel that you or your family would give misinformation, which is unfortunate, but normally people will respond to the questionnaire with a valid response.
      Questionnaires are structured to use a conversational method of asking people their opinions.
      Have you ever this or that?
      When you were this or that or thating did you have your dog with you?
      How often do you this or that?
      People are who do surveys are usually interesting and genuine in their I part of the process. Those who aren’t interested simply don’t participate.
      Opinion polls have been conducted forever and they will be conducted forever because people are interested in the general ‘feel’ in the community.
      Next time someone approaches you to be involved I do hope you are courteous and give of your time.
      You could change the world with your contribution.

      1. Anonymous2 months ago

        Bravo, Leslie!

  8. Anonymous2 months ago

    random sampling does not guarantee valid representative sampling, it does the best possible job of avoiding systematically biased samples.

  9. Anonymous2 months ago

    As a grad student in sociology, I thought it was a great video! Easy to understand for the general population. And, thanks, Scott, for your comments and suggestions!

  10. Scott Wolfel2 months ago

    Good video on polling basics, but the video should have also discussed populations, samples and margins of error, which are one of the most misunderstood aspects of polls.

    Most people assume that results are an accurate point estimate of a population, which is how they are usually reported, but in fact they’re estimates with a 3%-4.5% margin of error in both directions (with Type I and II errors). This means that, while polls can offer useful general information about the characteristics of a population, as with the recent election, the population can vary widely from the sample, and, while not the “likely” (>50%) outcome, Trump’s victory was well within the margin of error, particularly given the dynamic and rapidly changing nature of the population. Had people, and journalist, understood polls better, Trump’s victory wouldn’t have been such a shock to site just one example.

    Polls are a useful tool, offering a generally accurate model/map, but they are not the territory. While more complicated, personally I think polls should be reported with the margins of errors explicitly laid out, since most people don’t, and aren’t even able to, do the mental math of converting an estmate into a range. The video provides a useful introduction to why polls are useful in guaging public opinion, but it neglects explaining their important, and frequently misunderstood, limitations.

    1. joan murphy2 months ago

      This is only the first video of the series.