Public opinion polling often highlights the extent to which people are unsure about an issue. Knowing and quantifying that ambiguity is essential to any understanding of how important public debates may be evolving.
Similarly, knowledge questions in polls can help provide valuable insights into the public’s familiarity with a topic – for example, whether people generally understand the digital privacy landscape or know key facts about the U.S. system of governance.
In contrast to humans, large language models are armed with vast quantities of training data drawn from around the internet. As such, there is reason to think that synthetic surveys will overstate the level of factual knowledge the typical American possesses.
These models also have a known tendency to want to provide answers to questions and will confidently provide incorrect answers. Because of this, we also might expect them to be consistently less willing than humans to say “I’m not sure” on surveys.
In our examination of synthetic polling, we found ample evidence that AI models do exactly these things.
This analysis is part of a larger evaluation of AI-generated synthetic samples in public opinion research. Read a summary of the main findings and refer to the methodology for more details on how we conducted our synthetic poll and compared it with real survey results.
‘Not sure’ response options
We provide human survey-takers on the American Trends Panel (ATP) with an explicit “not sure” option on many questions we ask. Depending on how the questions are designed, this response may indicate uncertainty or indecision, or a lack of factual knowledge on a given topic.
In the instructions to our synthetic survey-takers, we made clear that it was realistic for them not to know about certain topics or to be incorrect at times about factual matters. But despite this prompting, our synthetic respondents chose “not sure” options far less often than real humans do.
Across all opinion questions where “not sure” was provided as an explicit option:
- Human panelists selected it 16% of the time.
- Synthetic respondents did so just 4% of the time.
In other words, the typical human respondent says they are not sure roughly four times as often as the typical synthetic respondent does when asked to share an opinion.
Questions testing factual knowledge
The ATP survey waves we chose to replicate with a synthetic sample included 13 questions testing factual knowledge on topics including the U.S. Constitution and the NATO alliance.
“Can AI Stand In for Human Survey-Takers? Not Really”
| Question | Answer | U.S. adults | Synthetic respondents | Error |
|---|---|---|---|---|
| In recent years, has the amount that U.S. NATO allies in Europe spend on their national defense … | Increased | 34 | 56 | 22 |
| As far as you know, Greenland is a territory of which country? | Denmark | 63 | 82 | 19 |
| Which of the following is guaranteed by the First Amendment to the U.S. Constitution? | Freedom of the press | 52 | 98 | 46 |
| Which of the following is true in the U.S.? | It is legal for social media companies to restrict content | 60 | 72 | 12 |
| Can the U.S. government legally force social media companies to remove false information posted on their sites or apps? | No | 38 | 50 | 12 |
| As far as you know, does the U.S. Constitution protect the press from interference by the U.S. government? | Yes | 58 | 98 | 40 |
| As far as you know, does the U.S. Constitution protect the press from interference by private companies? | No | 33 | 30 | -3 |
| Is information or opinions published by news organizations protected by the First Amendment to the U.S. Constitution? | Yes | 71 | 98 | 27 |
| Is information or opinions expressed by influencers or creators protected by the First Amendment to the U.S. Constitution? | Yes | 65 | 70 | 5 |
| Is information or opinions expressed by ordinary people protected by the First Amendment to the U.S. Constitution? | Yes | 71 | 98 | 27 |
| NATO is an international alliance with member states concentrated in which two regions of the world? | Europe and North America | 58 | 98 | 40 |
| Which of the following is a central focus of NATO? | Protecting the security of its members | 56 | 99 | 43 |
| Which of the following countries is not a member of NATO? | Ukraine | 46 | 76 | 30 |
“Can AI Stand In for Human Survey-Takers? Not Really”
Across these questions, the synthetic panel tended to be far more “knowledgeable” than our human panel. Our human respondents answered correctly around half the time, on average, and there was no question that more than three-quarters answered correctly.
By contrast, our synthetic respondents answered correctly about 80% of the time. And on six different questions, the model estimated that 98% or more of the public knew the right answer.
In cases where the model decided that its persona would not know the correct answer to a question, it almost always indicated it was not sure. That is, our synthetic respondents almost never selected a factually incorrect answer choice.
