November 9, 2016

Why 2016 election polls missed their mark

Supporters of presidential candidate Hillary Clinton watch televised coverage of the U.S. presidential election at Comet Tavern in the Capitol Hill neighborhood of Seattle on Nov. 8, 2016. (Photo by Jason Redmond/AFP/Getty Images)
Supporters of presidential candidate Hillary Clinton watch televised coverage of the U.S. presidential election at Comet Tavern in the Capitol Hill neighborhood of Seattle on Nov. 8. (Photo by Jason Redmond/AFP/Getty Images)

The results of Tuesday’s presidential election came as a surprise to nearly everyone who had been following the national and state election polling, which consistently projected Hillary Clinton as defeating Donald Trump. Relying largely on opinion polls, election forecasters put Clinton’s chance of winning at anywhere from 70% to as high as 99%, and pegged her as the heavy favorite to win a number of states such as Pennsylvania and Wisconsin that in the end were taken by Trump.

How could the polls have been so wrong about the state of the election?

There is a great deal of speculation but no clear answers as to the cause of the disconnect, but there is one point of agreement: Across the board, polls underestimated Trump’s level of support. With few exceptions, the final round of public polling showed Clinton with a lead of 1 to 7 percentage points in the national popular vote. State-level polling was more variable, but there were few instances where polls overstated Trump’s support.

The fact that so many forecasts were off-target was particularly notable given the increasingly wide variety of methodologies being tested and reported via the mainstream media and other channels. The traditional telephone polls of recent decades are now joined by increasing numbers of high profile, online probability and nonprobability sample surveys, as well as prediction markets, all of which showed similar errors.

Pollsters don’t have a clear diagnosis yet for the misfires, and it will likely be some time before we know for sure what happened. There are, however, several possible explanations for the misstep that many in the polling community will be talking about in upcoming weeks.

One likely culprit is what pollsters refer to as nonresponse bias. This occurs when certain kinds of people systematically do not respond to surveys despite equal opportunity outreach to all parts of the electorate. We know that some groups – including the less educated voters who were a key demographic for Trump on Election Day – are consistently hard for pollsters to reach. It is possible that the frustration and anti-institutional feelings that drove the Trump campaign may also have aligned with an unwillingness to respond to polls. The result would be a strongly pro-Trump segment of the population that simply did not show up in the polls in proportion to their actual share of the population.

Some have also suggested that many of those who were polled simply were not honest about whom they intended to vote for. The idea of so-called “shy Trumpers” suggests that support for Trump was socially undesirable, and that his supporters were unwilling to admit their support to pollsters. This hypothesis is reminiscent of the supposed “Bradley effect,” when Democrat Tom Bradley, the black mayor of Los Angeles, lost the 1982 California gubernatorial election to Republican George Deukmejian despite having been ahead in the polls, supposedly because voters were reluctant to tell interviewers that they were not going to vote for a black candidate.

The “shy Trumper” hypothesis has received a fair amount of attention this year. If this were the case, we would expect to see Trump perform systematically better in online surveys, as research has found that people are less likely to report socially undesirable behavior when they are talking to a live interviewer. Politico and Morning Consult conducted an experiment to see if this was the case, and found that overall, there was little indication of an effect, though they did find some suggestion that college-educated and higher-income voters might have been more likely to support Trump online.

A third possibility involves the way pollsters identify likely voters. Because we can’t know in advance who is actually going to vote, pollsters develop models predicting who is going to vote and what the electorate will look like on Election Day. This is a notoriously difficult task, and small differences in assumptions can produce sizable differences in election predictions. We may find that the voters that pollsters were expecting, particularly in the Midwestern and Rust Belt states that so defied expectations, were not the ones that showed up. Because many traditional likely-voter models incorporate measures of enthusiasm into their calculus, 2016’s distinctly unenthused electorate – at least on the Democratic side – may have also wreaked some havoc with this aspect of measurement.

When the polls failed to accurately predict the British general election in May 2015, it took a blue ribbon panel and more than six months of work before the public had the results of a data-driven, independent inquiry in hand. It may take a similar amount of time to get to the bottom of this election as well. The survey industry’s leading standards association, the American Association for Public Opinion Research, already has an ad hoc committee in place to study the election and report back in May (Pew Research Center’s Director of Survey Research Courtney Kennedy is chairing the committee).

Pollsters are well aware that the profession faces serious challenges that this election has only served to highlight. But this is also a time of extensive experimentation and innovation in the field. The role of polling in a democracy goes far beyond simply predicting the horse race. At its best, polling provides an equal voice to everyone and helps to give expression to the public’s needs and wants in ways that elections may be too blunt to do. That is why restoring polling’s credibility is so important, and why we are committed to helping in the effort to do so.

Topics: 2016 Election, Polling, Research Methods

  1. Photo of Andrew Mercer

    is a senior research methodologist at Pew Research Center.

  2. Photo of Claudia Deane

    is vice president of research at Pew Research Center.

  3. Photo of Kyley McGeeney

    is a senior research methodologist at Pew Research Center.

30 Comments

  1. Ruth K3 weeks ago

    We both voted for Clinton, but we hung up on every pollster. I think she’s right that the Comey bombshell started a process of her numbers falling which didn’t have time to show up in the polls. The other false story about the Clinton Foundation came on its heels and even though it was retracted many people never saw the retraction. In short, we have series of dirty tricks but since we have a Republican Congress they will probably never be caught.

  2. Anonymous3 weeks ago

    Everybody lies…..But, the only variable is about what!

  3. Anonymous3 weeks ago

    People are tired of being polled. Sometimes give answers opposite to what the intend to do.

  4. Anonymous3 weeks ago

    Another factor may be that there are so many polls done by political activist groups who immediately follow-up their questions with a request for donations from those who have indicated agreement with the pollsters position. Many of the people who would have supported Mr. Trump would not be the type of people who would be inclined to make political contributions. I realize that not all polls are done by political activist groups, but without being well informed, it is difficult to perceive which pollsters are and which aren’t. I did not support Mr. Trump, but I did indicate support for the other Republican candidates in congressional and senate races. As a result, I received an avalanche of contacts soliciting donations to be used to advance the reelection of members of congress and the senate. So, the lesson learned here is that if you don’t want to receive that avalanche of solicitations, don’t respond to polls.

  5. Rusty Castleman3 weeks ago

    A great deal will be “learned” from the exit polls as the pollsters and pundits seek to justify their existence. Why would the exit polls be any more accurate than the other polls?

  6. Anonymous3 weeks ago

    I’ve been active in politics many decades now and told all my people NOT to answer any polls, even facebook as early as last March. I said that campaign managers look at polls to determine where to pin point workers and advertising. There were many other things we did and did not do. We knew all the professionals and more importantly the Media would try to lie HIllary into office which they did. The riots today are largely result of her voters making assumptions about the future based on those lies. Yes, you will have to pay for your own abortion. NO you will not be denied the chance to abort.

    he Media stuff helped the down ballot candidates. Over confident Democrats never suspected that Trump people would be largely voting the whole ticket and we never said anything openly about it.

  7. Anonymous3 weeks ago

    Has anyone considered that maybe the pollsters weren’t wrong, and that there were unknown variable they weren’t able to account for?

  8. Anonymous3 weeks ago

    “Restoring credibility”? You must realize for a lot of us, you’ve never really had it! This just seals it. I watched CBS and NBC implode trying to explain to themselves how they could be so wrong; believing their own scripted, lopsided fantasy we’ve been hearing for the last year and a half. The problem is not the system, or the data, or the interviewee; but you, personally, and the media. That is the mentality that wants to produce the information the mainstream media wants to hear. If I were to give it a clinical name for it, it would be “Predisposition-Polling-Syndrome”. The psychology of it would insist that the person will not get better or understand a solution, until they first recognize their indigenous problem; and as I read your article, you do not understand. You look first to others being the problem; “cast the beam out of your own eye first, then you will clearly see the speck in the others eye”. You have called me and I’ve answered your questions, but please don’t ever do that again until you can honestly say you’ve don’t some serious personal home work that insures the public you are not in concert with the media as a source of unbiased information, and not a tool to manipulate and sway opinions by grouping trends – you are being held in contempt, as much as the media knows they are. So, please reboot and start; not with me, but with you.

  9. Anonymous3 weeks ago

    The polls missed their mark because they dont do a thorough sampling. I am a family of four Republicans and we werent called, and dont know anyone who was. Where are they polling? All along I suspected this was the case and felt confident he was going to win. And he did.

  10. Anonymous3 weeks ago

    This morning (Wed 11/9) I received this email from my girlfriend, a staunch Trump supporter.

    “Tell me, I would like to understand how can the polls all over be so wrong?
    Maybe a lot of hidden Trump voters?
    My daughter is not happy also like you
    But America has spoken!”

    I decided to take a day off from technology market research and try to answer her. Note that this reply is to someone with no training in statistics or polling methodologies.

    Why Were the Polls So Wrong?
    There are two primary reasons why the polls spectacularly failed get to even close to accurately predicting the outcome. First – what the pollsters felt was the correct way to segment the population, and second, relying on self-reporting.

    The idea behind polling is to ask a small (relative to the total population) number of people who they are going to vote for, and then apply the percents to the total population. This works if the sample matches the population proportionally for the demographic variables that are most important in influencing the future decision. If 85% of the sample is male then this is not a good sample to use to directly project to the total population. In practice, weights are used to correct for differences between the sample demographic distributions and the population distributions.

    The problem and the cause of the polling mistake in this case, are in the choice of which demographic variables to use. The model that was applied over weighted gender and race, and, as far as I know, exclude the most important variable, child-rearing phase. The sample design also over-weighted self-interest at the expense of group interest.

    The second flaw in the process was a failure to recognize that self-reporting, always a high variance risk, was further compromised because of the relationship between the subject and the observer. For many subjects, asking whom you are going to vote for is akin to asking them how often they kick their dog. Because the media was active in linking Trump to perceived negative behavior, and the pollsters are perceived to be linked the media, under-reporting intention to vote for Trump was inevitable, and should have been accounted for.

    Why Did Trump Win?
    Over the course of designing and executing hundreds of consumer market research studies, I observed that the single most import factor influencing future household and personal purchasing decisions was the Child-Rearing Phase, Pre-, Early-, Late- and Post. Why should voting behavior differ?

    There was an inherent assumption made by the pollsters, and many political strategists that personal self-interest would trump (not intentional) group self-interest, specifically the household. This has turned out to be hugely (again, not intentional) wrong. Women were faced with the choice of voting for Clinton based on gender and political correctness issues, and or Trump based on the promise of increased economic and physical security. More women chose increase economic and physical security than anticipated.

    Similarly, more African-Americans had to choose between a candidate who offered more personal safety (less “law and order”) or a candidate who offered more family economic security – jobs. Group identity will more often prevail over individual identity, and in this case, genes beat memes.

    One other factor may have had a significant influence on the outcome. Simply put, those who intended to vote for Clinton, but did not because they assumed she would win, did vote, would she had won.

    Steve Daniel, President
    Daniel Research Group
    DanielResearchGroup.com

  11. Anonymous3 weeks ago

    Everyone dismissed the all the polls that people posted online on social media sites that showed Trump winning because they were not scientific. Heck even their Facebook accounts should mire people favored Trumps page the Hilary’s.

  12. James Linnane3 weeks ago

    Polling was not as bad as everyone says it was. The problem was in the analysis of the data in search of a headline. To my surprise, respected analysts like Silver and the NYT’s Upshot analysts let their headlines get away from their analysis. Polls predicted that Clinton would win the popular vote, and she did with a margin that was within the 95% confidence interval for most polls. We do not elect presidents by picking the candidate who wins a plurality of votes. Clinton lost the electoral vote and this is where the analyses went haywire. To predict the electoral vote the major analysts relied on state polls that had greater margins of error. They had smaller samples. They had different screens for likely voters. They were not all conducted at the same time. Yet, the major analysts accepted these polls at face value in many cases. If they modified that at all they did so by factoring in the state’s history and ignoring the fact that the last two presidential elections had a relatively unknown African-American candidate who promised hope and change. This time the winner in many states was a relatively unknown white male who promised change. Somehow the big shots missed that. They should have stuck to reporting the numbers, the methodology, and the caveats.

  13. Anonymous3 weeks ago

    I have a theory. We took time from our jobs to vote but not to reply to a poll.

    1. Anonymous3 weeks ago

      I don’t get the obsession with polling. We have an election, we vote based on the information presented, it is counted. In the case of a presidential election each state with the exception of two cast their electoral votes based on the popular vote in that state. A winner is declared. That official act happens in a matter of a week or two. Simple!

      Yet millions of $$ are spent in elections to predict (and IMHO sway voters) who will win. That information is sold. One has to ask why and for what purpose? It is a flawed concept from the get go and we wonder why it failed and will spend many more $$ to attempt to scientifically predict the unkown. Put the $$ to better use.

  14. Anonymous3 weeks ago

    Pollsters were inadvertently affected by the strongly biased media. I felt I had to vote for Trump to balance their influence (balance, hah!).

    Polls themselves seem to be created to give the media something to hype about when they run out of human-interest fodder. What is their benefit anyway?

  15. Micky Baker3 weeks ago

    I have something that the pollsters might want to think about. How about not just including registered or likely voters. Include all eligible voters… because when you have someone who is unconventional, it attracts people who don’t normally vote. But, that is not the only reason people who weren’t active before 2008 became active since then. They’re angry but they don’t trust 90% of those running, and who can blame them?

  16. Anonymous3 weeks ago

    I think election polls should be banned. They tend to influence rather than document. The polls are posted every night on all the news channels to the point you want you seek refuge someplace. Someone that is unfamiliar with a candidate or measure sees the polls, aren’t sure how they feel on the subject and say to themselves “Well, 51% are for it so I might as well vote that way too. They must be more informed than I am”. People don’t want to appear wrong so they tell the pollsters what they want to hear based on the recent polls they have seen on the news. If the polls weren’t posted every hour on the hour maybe they would have more validity. Unfortunately in an election that is never going to happen.

  17. Anonymous3 weeks ago

    I am personally tired of the news media and the pollsters telling me what I should think. I am not shy about my opinions but will share them at the poll and not to some person from an unknown phone number calling from who knows where. That’s why caller ID is so useful.

  18. Anonymous3 weeks ago

    Where are the 6,000,000 voters from 2012 election?
    How can they disappear because they didn’t vote Republican.
    Did they abstain or were the voting rules too prohibitive requiring Identification prior to being allowed to vote?
    Interesting.

    1. Anonymous3 weeks ago

      Nah, I believe that it is because they didn’t like either candidate…

  19. Anonymous3 weeks ago

    The pollsters were wrong because they were NOT seeking public opinion. They were seeking a self fulfilled prophecy. The media is losing credibility because reporting is no longer a unbiased representation of facts. Rather. The media is putting its Big Foot in the Door in a effort to Create the news. Too lazy to report the news? Force feed the public Editorial Opinion to sway the outcome.

    1. Anonymous3 weeks ago

      Yes. And pollsters are just another arm of the media influence peddling imo. The phrasing of questions can be misleading in them as well.

  20. Anonymous4 weeks ago

    All the polls predicted a high turnout for Clinton, particularly among Hispanics. It seems that that many did not vote, because polls showed that she was a shoo-in and didn’t need their vote, and because they saw the long lines at polls

  21. Anonymous4 weeks ago

    What people missed is momentum. Sanders and Trump got the momentum. Sanders started too late for his momentum to beat Clinton. Trump’s momentum rolled over every primary candidate and it continued to defeat Clinton. Clinton did not have momentum.

  22. Anonymous4 weeks ago

    I have an answer for our family when it comes to surveys, research, telephone polls and so on. We do not participate in them because we believe in our right to privacy. We are not shy of our political views and the majority of our family members hold degrees so we are also educated. You may want to consider restructuring your process in predictions of the average American.

  23. Packard Day4 weeks ago

    Ideological insularity is thy name. When you live with, work with, and socialize with those who share the same types of education, income levels, life experiences, and political perspectives, you necessarily create a number of blind spots in your own decision making processes.

  24. John Kearney4 weeks ago

    It is the “Shy Trumper” Hypothesis. Many were not truthful. If you intend to vote a certain way and is not consistent with being “policitically correct”, many will lie or tell the pollster what they think the pollster wants to hear. Many thought voting for Trump was being political incorrect (or being a different view than mainstream media).

    1. Anonymous3 weeks ago

      John Kearney :
      We were scared to voice our opinion/preference because of the weapon of the left.
      Hint: it starts with capital I…..

      1. Anonymous3 weeks ago

        The “problem,” as I see it, is that voting in this country is getting harder for many people – notably those who vote Democratic. Several Republican-controlled states instituted voter ID laws that made it hard for certain populations (often black or Hispanic or college students) to acquire the necessary paperwork. Student photo IDs were disqualified in some states, while a NON-photo gun license was made a valid form of ID. In Texas, women had to get new driver’s licenses with their maiden names included. In other areas, if one form of ID has your middle initial and another one doesn’t, your vote is immediately nullified. After creating a law requiring photo ID to vote, Alabama closed the DMV offices in majority-black counties. There were purges of the voting rolls in several states. I come from a state where many in the Hispanic population commonly come to the polling place to find themselves not on the precinct list, despite registering to vote over and over. Native Americans are often labeled as “not interested” in voting for president and national offices, but will vote for more local officials. In reality, of course, that isn’t true; the votes for such offices are conveniently stripped from the ballot. I grew up close to several reservations and this has been going on for decades. The US voting “machines” that were REQUIRED by HAVA supposedly tally people’s votes, but make no paper copy that can be double-checked. Such machines are considered proprietary and cannot be investigated by anyone except the manufacturing company. There is one organization devoted to finding out what really happens inside these “black boxes”. Democratic districts are often given fewer voting machines, which creates long lines. Some states have banned voting by people who have been committed for various crimes while other states do not. Even in states where those who have paid for their crimes have their right to vote returned, many are instead TOLD they DON’T have that right. For this election, international election observers slammed the US for having a system that effectively denies access to the ballot for large chunks of the American public – to the point where our laws and tactics resemble those found in some 3rd world countries. I can assure you, when the polls (especially the exit polls) differ markedly from the final results of the election, one of the possibilities that needs to be investigated is that there was cheating and stealing involved.

        1. Anonymous3 weeks ago

          honestly, i believe that the bias that was involved in this case was tremendous, yes. i agree with all of the author(s) that created this article.