July 28, 2014

Q/A: What the New York Times’ polling decision means

The New York Times and CBS News made big news in the polling world this weekend when they announced that they will begin using online survey panels from YouGov as part of their election coverage. YouGov, a U.K.-based research firm founded in 2000, uses such panels rather than traditional telephone surveys; the panel the Times and CBS are using has more than 100,000 members. The Times, citing concerns about the dearth of high-quality, non-partisan survey data, particularly at the state level, says it plans to include YouGov results as part of “a diverse suite of surveys employing diverse methodologies.”

Keeter
Scott Keeter, the Pew Research Center’s director of survey research

While panels have long been used by market researchers, they’re relatively new in the opinion-research field, and views on them are sharply divided. We asked Scott Keeter, the Pew Research Center’s director of survey research, to explain the issues at stake and give us his preliminary thoughts.

What’s different about the YouGov panel surveys from the surveys previously used by the Times, CBS and Pew Research?

There are two big differences. One is that these are conducted entirely online, among internet users. People who don’t use the internet aren’t included (more on this below). The other – and arguably the biggest — difference is that the samples for these surveys are selected using so-called non-probability sampling methods. For decades, only probability – or random – samples have been generally accepted as a scientific way to produce accurate, representative samples for surveys.

Explain the difference between a probability and a non-probability sample.

The American Association for Public Opinion Research, the leading association of survey research professionals, has explained it well. Here’s how their Task Force on Non-Probability Sampling put it in their major report last year: “In a probability sample, everyone in the population of interest (e.g., all registered voters in a political poll) has a chance of being selected for an interview. Knowing those chances is critical to creating valid statistical estimates.” Non-probability samples, in contrast, “are those in which the participants are chosen or choose themselves so that the chance of being selected is not known.”

Here’s how the Times’ Nate Cohn explained how YouGov puts its online panel together: The firm “attempts to build a large, diverse panel and then match[es] its panelists to demographically similar respondents from the American Community Survey….This step is intended to mimic probability sampling. But it can require significant assumptions about the composition of the electorate, including partisanship.”

The biggest benefits of probability sampling are (1) the fact that the researchers are selecting respondents, rather than allowing respondents to volunteer to participate, and (2) that knowing the chances of selecting an individual allows us to estimate how much any given sample is likely to differ from what we would have found had we interviewed everyone in the population. That shows up as a survey’s “margin of error.” There is no comparable margin of error for non-probability samples.

So why would the Times and CBS look to non-probability methods? Do probability-based methods such as random-digit dialing (RDD) telephone surveys have their own problems?

Indeed they do. As our previous research on survey non-response has shown, it’s increasingly difficult to reach and interview people. But there are two important points to make. First, telephone polls – including ours — did a generally good job predicting elections in 2012 and in previous years. Our samples continue to look very much like the population on most indicators for which we have reliable data from the Census and other high-quality sources.

Second, the troubles faced by one method of research cannot, by themselves, be used to justify the adoption of an alternative. The alternative has to prove itself to be accurate enough and precise enough for the purposes to which we currently apply RDD telephone surveys. The field knows a lot about the biases and problems with our current methods. There is a much less complete understanding of the biases and problems with non-probability methods, but I’m hopeful that we’ll gain that understanding through a process of careful experimentation

What does it mean for journalism, and for the survey world more generally, that the Times and CBS, of all news organizations, have decided to use YouGov’s panel? Would you expect other news organizations to follow now that those two have broken the ice?

This is a very big deal in the survey world. Until now, no major news organization has put its brand on using surveys based on non-probability methods. The move has set off a very lively debate on Twitter among journalists and pollsters. There are strong opinions about the issue of non-probability samples. Here’s one view, from Washington Post pollster Scott Clement:

And a different view from Amy Walter, national editor of the Cook Political Report:

I can’t predict what other organizations are going to do, but I do expect this to spur more experimentation – and that’s a good thing for the field. Because the Times and CBS News have good reputations for transparency, I fully expect that we will learn a lot more about the YouGov methodology in the coming weeks. That’s a good thing as well.

How can a survey based only on internet users provide an accurate representation of the entire public?

It’s worth remembering that not everyone is online: According to our most recent estimates, 89% of U.S. adults use the internet. The good news from a polling perspective is that that figure is steadily increasing – it was 79% just five years ago – so the potential bias from excluding non-internet users in getting smaller and smaller.

It’s still the case that people who don’t use the internet are different in many ways from those who do – in particular, they are older, poorer and less educated. But their dwindling numbers mean that their absence from a survey won’t make a huge difference in the findings on most questions. Still, we think it’s important to be able to describe our samples as “nationally representative” and try to make sure that they are whenever possible.

If this proves to be a successful endeavor for the Times and CBS News, does that mean other pollsters will embrace non-probability sampling?

Not necessarily. It’s important to keep in mind that online non-probability panels vary in quality, just as probability-based surveys do. One of the most important points in the AAPOR Task Force report is that there’s no single consensus method for conducting “non-probability sampling.” There are many different approaches, and most of them don’t have the public record of performance that YouGov has. YouGov has been conducting public polls in elections for many years. As a result, they have a track record that can be compared with probability-based polls. Until we have more organizations conducting polls in advance of elections and explaining their methods in detail, I believe that adoption of non-probability sampling for political polling will proceed slowly.

Isn’t Pew Research using an online panel right now?

We do have a panel – it’s called “The American Trends Panel” — but it’s very different from the one that the Times and CBS are using. It’s based on a probability sample, and while most of the interviews are conducted online, we also have panelists who don’t use the internet. We interview those individuals by mail or phone. Here’s a link to more detail.

Is Pew Research ever going to use the kind of online non-probability panel that the Times and CBS are using?

Yes, we will – but the real question is what we will use it for. Our current standards permit the use of non-probability samples for certain purposes, such as conducting experiments or doing in-depth interviews. In addition, we have embarked on a program of research to help us better understand the conditions under which non-probability samples can provide scientifically valid data. We also are exploring how to utilize non-survey data sources, which by their very nature tend to come from “samples” that are not random. But until we understand the pros and cons of those methods a lot better, we’re going to be very cautious about incorporating them into our research.

Topics: Polling, Research Methodology

  1. Photo of Drew DeSilver

    is a Senior Writer at the Pew Research Center.

Leave a Comment

Or

All comments must follow the Pew Research comment policy and will be moderated before posting.

20 Comments

  1. lindaj3 months ago

    Interesting ideas here. But, you don’t mention the percent of people who are left out of random surveys because of no land line phones and those who won’t answer a mobile from an unknown caller. It may our may not give good results, we have to wait to see. I got 4 real polling calls last election season. As an old person with a land line, NOT A GOOD SAMPLE!
    But even I am seriously considering going mobile only…

    Reply
  2. John Danahy3 months ago

    I’ve never been polled by PR or NYT. I have confidence both will have the highest degree of having specific comment coming under someone’s eyeballs. I feel most surveys are a waste of time because I am very suspect of the sponsor, the political partisanship, and accurate use of my input.

    With the upcoming election season BS fest, I’d certainly appreciate being a part of both sources, but I don’t know how to do so or if selection is merely the luck-of-the-draw.

    Reply
    1. lindaj3 months ago

      Yes, I hate wading time on push polls, and hang up as soon as it’s obvious. Which isn’t always right away. The 4 I mentioned in my comment were real, didn’t count the fakes. You are right on.

      Reply
  3. C Kirk+Osterland3 months ago

    a good move.Provides a larger and more varied base,almost like a referendum.I don’t know what % of voters participate in a referendum? cko

    Reply
  4. Mike Hihn3 months ago

    Is there a known equivalent to the percentage of people who use the Internet — meaning what percentage can even be reached by phone these days?

    On that percentage basis, I’m guessing the Internet is higher overall, with a concern on how critical it may be to “exclude” non-Internet users. Are they overwhelmingly or largely an entire class being excluded, or fairly neutral to a larger American profile?

    (I’m on three online polling panels, all consumer oriented)

    Reply
  5. Nicholas P. Schiavone3 months ago

    Requires very careful methodological research and professional auditing (e.g., E&Y). Engage the Media Rating Council for counsel and support services.
    Quality of understanding and decision-making is at stake in the American Democracy.

    Reply
    1. David Lloyd-Jones3 months ago

      Nick,

      The across-the-board corruption of the Big Eight, now down to three or four, accounting firms’ audit practices is one of the reasons for the present parlous state of (your caps) American Democracy.

      It’s not just the economic crashes of ’08, ’91, ’87 with their roots in dishonest business with GAAP stamped all over it. America is in as bad shape in the good years as it is in the years when the rot surfaces as a crash subsequently bailed out of. The corruption is routine.

      Your Media Rating Council may be a better suggestion. Unlike companies, which use and pay auditors to help them fool their shareholders, their banks, and themselves, your Council seems to be paid by an industry full of competing interests. As long as they avoid the accounting industry format — where firms pay for their own good reviews — there may be some hope for it. Your “professional accounting”? Not so much.

      -dlj.

      Reply
  6. Richard3 months ago

    okay then.

    Reply
  7. Richard3 months ago

    This method of polling is the precursor of local and national direct primaries (that Truman always wanted), general election and within a century, concluding –as in The End, finis, fin and ende — of legislatures. Pure Democratic Republic (cum anarchy?)

    Reply
  8. Pete Noffke3 months ago

    Polls or surveys are generally frustrating as they seldom give as an option a view that I have. They don’t offer an op-ed option. Opinions vary from person to person. When I read results of polls or surveys quite often I’m amazed at what I’m reading thinking how can people possibly feel that way. So to change the way it’s done I really would like to know what the goal is politically.

    Reply
  9. humboldt3 months ago

    Has anybody thought about a whole bunch of folks that never tune into CBS, or read the NYT?

    Reply
  10. Natalie Davis3 months ago

    One issue to be addressed is whether there are differences in how candid a respondent is on the telephone vs being polled online. My experience is that online respondents are less likely to offer socially desirable answers. Online also reduces interviewer bias and variability. Weighting always has problems, but on balance I like an online methodology.

    Reply
    1. Natalie Davis3 months ago

      Not sure I understand why this post needs to be moderated.

      Reply
      1. Mike Hihn3 months ago

        I’ll defend PEW on this. If one spends a lot of time in online forums, as I do, one soon sees abusers — called trolls — even where you’d least expect them.

        There’s also ads like, “My brother makes 50 gazillion dollars per week by simply polishing his own shoes! Click this link for details.”
        Those are more annoying than offensive, but they can be overwhelming at times.

        Reply
  11. Troy Carter3 months ago

    Why, in all the reporting done by CBS, the times, and yougov on their first panel response, not indicate the time frame in which panelists had to answer? Which I understand to have been from July 5 to July 24 — this is a rather large time period for a poll – an answer given on July 5 could easily change by July 24- esp when scandals break– I’m still skeptical – I think this method is problematic – Troy Carter, Bozeman Daily Chronicle

    Reply
  12. Poniesinjudah3 months ago

    Ironic on this topic your comment sign in doesn’t include twitter. Anyway in Toronto for the last 4 months or so the only polls on the mayoral election featuring yes, the crack mayor (who was constantly taped falling down drunk in public, went to rehab in May but didn’t complete it) have been these non-probability online ones. The crack mayor keeps polling around 30%. One of the other mayoral campaigns claimed to a reporter to have internal polling showing crack mayor Rob Ford at 16%. So on Oct 27th we’ll see if the online poll thing was wildly out. If those polls stay around 27% to election day and Ford then gets less than 10% of vote that would really tell us something.

    Reply
  13. Cathy Grossman3 months ago

    How will we be able to report accurately trend data when there is a methodology switch between earlier phone-based surveys and recent panel-based surveys?

    Reply
    1. Scott Keeter3 months ago

      Great question, Cathy. The major reservation most people have about online panels — especially those based on non-probability samples — is that the sample itself is different from the sample used for a telephone survey. Your question goes to the other big concern: does the mode of interview have an impact on the results we get from otherwise identical questions? A lot of research has been done on this and the answer is generally “yes,” though the effects of mode is bigger for some kinds of questions than others. I’ll be writing more about this in a future post, but we are in the midst of a major experiment to study mode of interview effects. We are testing a broad range of questions from all of the research areas here (including many on religion) to see which ones are most likely to be affected by a switch from telephone to self-administration. In the meantime, we are being very cautious about how we are using the American Trends Panel. Most of the questions we are reporting from it are new questions that have no trend. After we complete our study of mode effects, we may decide that some questions are relatively unaffected by the mode of interview, and we’ll feel comfortable trending them to our telephone surveys. But I’m certain that some questions we use will be affected by the difference in mode, and we will have to figure out a way to report those with appropriate caveats so that readers won’t be misled when looking at trends over time.

      Reply
      1. Thomas Riehle3 months ago

        Scott, you are doing our poor, troubled polling industry a great service by leading Pew’s investigation of this and publicizing your thoughts so transparently. Your disclosure here of Pew’s intention to thoroughly investigate effects of interviewing mode on results is great news. I am especially glad that AAPOR, in its largely unhelpful letter on this topic today, included a link to this analysis from you for those who want to read more. Thanks!

        Reply
      2. Dot3 months ago

        There many of us that refuse to respond to polls. How does that affect the results?

        Why don’t you and other media simply report the results of “established” polls with how they were determined and let the public see for themselves how they compare with reality?

        Reply