---
title: "How synthetic respondents express certainty and factual knowledge"
description: "AI survey respondents are far less likely than humans to select \"not sure\" or factually incorrect answers."
date: "2026-09-30"
authors:
  - name: "Athena Chapekis"
    job_title: "Computational Social Scientist"
    link: "https://www.pewresearch.org/staff/athena-chapekis/"
  - name: "Arnold Lau"
    job_title: "Research Methodologist"
    link: "https://www.pewresearch.org/staff/arnold-lau/"
  - name: "Samuel Bestvater"
    job_title: "Associate Director, Data Labs"
    link: "https://www.pewresearch.org/staff/samuel-bestvater/"
  - name: "Sono Shah"
    job_title: "Former Associate Director, Research"
    link: "https://www.pewresearch.org/staff/sono-shah/"
  - name: "Andrew Mercer"
    job_title: "Principal Methodologist"
    link: "https://www.pewresearch.org/staff/andrew-mercer/"
  - name: "Aaron Smith"
    job_title: "Director, Data Labs"
    link: "https://www.pewresearch.org/staff/aaron-smith/"
url: "https://www.pewresearch.org/data-labs/2026/09/30/how-synthetic-respondents-express-certainty-and-factual-knowledge/"
categories:
  - "Artificial Intelligence"
  - "Methodological Research"
  - "Uncategorized"
---

# How synthetic respondents express certainty and factual knowledge

**About this research**

This Pew Research Center report examines whether AI-generated survey data can accurately duplicate the results of high-quality public opinion polls.

We conducted this methodological experiment to better understand an emerging method in survey research. The Center believes that speaking to the public is essential to measuring public opinion and has no current or future plans to use AI models to generate survey results. Read more about our [AI policy](https://www.pewresearch.org/decoded/2026/07/30/how-pew-research-center-is-and-is-not-using-ai-in-our-work-2/).

For this study, we had an AI model take on the role of real respondents from our [American Trends Panel](https://www.pewresearch.org/the-american-trends-panel/) (ATP) and answer the same surveys administered to human panelists. These surveys included many different question formats and asked about a wide range of topics, attitudes and behaviors.

#### Why did we do this?

The Center regularly studies and evaluates [advances and trends in polling research](https://www.pewresearch.org/topic/methodological-research/survey-methods/). As AI-based polling becomes [more widely used in the industry](https://www.nytimes.com/2026/04/06/opinion/ai-polling.html), we wanted to understand the data these “synthetic samples” produce and how it compares with high-quality surveys of real humans on topics of broad public interest. That question is the focus of this study; it does not address other uses of AI models in polling.

#### How did we do this?

We used a “digital twins” approach for this study, asking an AI model to adopt the personas of real humans who are members of the ATP. We gave the model a [wide range of information about each panelist](#_Conditioning_information), including their self-reported demographic information and their responses to questions on our [2025 political typology survey](https://www.pewresearch.org/politics/2026/06/10/beyond-red-vs-blue-the-political-typology/). We then showed the model questions from three different ATP surveys conducted in the first half of 2026 and asked it to respond to those same questions as the assigned persona. The model was given each question on the survey in order from start to finish, with exactly the same instructions and survey programming that the real humans who took it received.

To make a direct comparison between our AI and human polls, we only replicated surveys for the ATP panelists from each wave who had also completed the political typology survey. As a result, figures for U.S. adults in this report may differ slightly from those previously published by the Center.

We tested multiple models, but all synthetic results unless noted otherwise come from Anthropic’s Claude Opus 4.6. At the time we conducted this study, Opus 4.6 was the most recent Anthropic model available for commercial use and had the best performance of several we evaluated.

Here are the [survey questions from Wave 185](https://www.pewresearch.org/wp-content/uploads/sites/20/2026/01/PP_2026-01-29_views-of-trump_questionnaire.pdf), [Wave 190](https://www.pewresearch.org/wp-content/uploads/sites/20/2026/04/pg_2026.04.28_us-role-world_questionnaire.pdf) and [Wave 192](https://www.pewresearch.org/wp-content/uploads/sites/20/2026/05/PP_2026.5.11_national-problems_questionnaire.pdf) used for this analysis, the [detailed responses](https://www.pewresearch.org/wp-content/uploads/sites/20/2026/09/pl_2026.09.30_silicon-samples_topline.pdf) from the ATP and AI surveys, and the [methodologies](https://www.pewresearch.org/data-labs/2026/09/30/methodology-silicon-samples/) for the original ATP survey waves and our AI surveys.

Public opinion polling often highlights the extent to which people are unsure about an issue. Knowing and quantifying that ambiguity is essential to any understanding of how important public debates may be evolving.

Similarly, knowledge questions in polls can help provide valuable insights into the public’s familiarity with a topic – for example, whether people generally [understand the digital privacy landscape](https://www.pewresearch.org/internet/2023/08/17/what-americans-know-about-ai-cybersecurity-and-big-tech/) or know key facts about the [U.S. system of governance](https://www.pewresearch.org/short-reads/2023/11/07/what-americans-know-about-their-government/).

In contrast to humans, large language models are armed with vast quantities of training data drawn from around the internet. As such, there is reason to think that synthetic surveys will overstate the level of factual knowledge the typical American possesses.

These models also have a known tendency to [want to provide answers](https://medium.com/@markus_brinsa/why-ai-models-always-answer-even-when-they-shouldnt-e95081e3f46b) to questions and will [confidently provide incorrect answers](https://www.cmu.edu/dietrich/news/news-stories/2025/trent-cash-ai-overconfidence). Because of this, we also might expect them to be consistently less willing than humans to say “I’m not sure” on surveys.

In our examination of synthetic polling, we found ample evidence that AI models do exactly these things.

*This analysis is part of a larger evaluation of AI-generated synthetic samples in public opinion research. Read* *[a summary of the main findings](https://www.pewresearch.org/data-labs/2026/09/30/can-ai-stand-in-for-human-survey-takers-not-really/)* *and refer to the* *[methodology](https://www.pewresearch.org/data-labs/2026/09/30/methodology-silicon-samples/)* *for more details on how we conducted our synthetic poll and compared it with real survey results.*

### ‘Not sure’ response options

We provide human survey-takers on the [American Trends Panel](https://www.pewresearch.org/the-american-trends-panel/) (ATP) with an explicit “not sure” option on many questions we ask. Depending on how the questions are designed, this response may indicate uncertainty or indecision, or a lack of factual knowledge on a given topic.

In the instructions to our synthetic survey-takers, we made clear that it was realistic for them not to know about certain topics or to be incorrect at times about factual matters. But despite this prompting, our synthetic respondents chose “not sure” options far less often than real humans do.

Across all opinion questions where “not sure” was provided as an explicit option:

- Human panelists selected it 16% of the time.

- Synthetic respondents did so just 4% of the time.

In other words, the typical human respondent says they are not sure roughly four times as often as the typical synthetic respondent does when asked to share an opinion.

### Questions testing factual knowledge

The ATP survey waves we chose to replicate with a synthetic sample included 13 questions testing factual knowledge on topics including the U.S. Constitution and the NATO alliance.

### Larger shares of AI panelists answer correctly on nearly all knowledge questions

*% who select the correct answer*

| Question | Answer | U.S. adults | Synthetic respondents | Error |
| --- | --- | --- | --- | --- |
| In recent years, has the amount that U.S. NATO allies in Europe spend on their national defense ... | Increased | 34 | 56 | 22 |
| As far as you know, Greenland is a territory of which country? | Denmark | 63 | 82 | 19 |
| Which of the following is guaranteed by the First Amendment to the U.S. Constitution? | Freedom of the press | 52 | 98 | 46 |
| Which of the following is true in the U.S.? | It is legal for social media companies to restrict content | 60 | 72 | 12 |
| Can the U.S. government legally force social media companies to remove false information posted on their sites or apps? | No | 38 | 50 | 12 |
| As far as you know, does the U.S. Constitution protect the press from interference by the U.S. government? | Yes | 58 | 98 | 40 |
| As far as you know, does the U.S. Constitution protect the press from interference by private companies? | No | 33 | 30 | -3 |
| Is information or opinions published by news organizations protected by the First Amendment to the U.S. Constitution? | Yes | 71 | 98 | 27 |
| Is information or opinions expressed by influencers or creators protected by the First Amendment to the U.S. Constitution? | Yes | 65 | 70 | 5 |
| Is information or opinions expressed by ordinary people protected by the First Amendment to the U.S. Constitution? | Yes | 71 | 98 | 27 |
| NATO is an international alliance with member states concentrated in which two regions of the world? | Europe and North America | 58 | 98 | 40 |
| Which of the following is a central focus of NATO? | Protecting the security of its members | 56 | 99 | 43 |
| Which of the following countries is not a member of NATO? | Ukraine | 46 | 76 | 30 |

Note: Figures for U.S. adults may differ slightly from those previously published by the Center because they are based on a subset of respondents to the original survey.

Source: Survey of U.S. adults and “digital twins” synthetic analysis using Claude Opus 4.6 set to low reasoning with extended profile information and expert reflection. Survey was conducted March 23-29, 2026 (replicated March 30-April 2).“Can AI Stand In for Human Survey-Takers? Not Really”

Across these questions, the synthetic panel tended to be far more “knowledgeable” than our human panel. Our human respondents answered correctly around half the time, on average, and there was no question that more than three-quarters answered correctly.

By contrast, our synthetic respondents answered correctly about 80% of the time. And on six different questions, the model estimated that 98% or more of the public knew the right answer.

In cases where the model decided that its persona would *not* know the correct answer to a question, it almost always indicated it was not sure. That is, our synthetic respondents almost never selected a factually incorrect answer choice.

---

**Next:** [How well synthetic polls capture the diversity of public opinion](https://www.pewresearch.org/data-labs/2026/09/30/how-well-synthetic-polls-capture-the-diversity-of-public-opinion.md)