--- title: "How well synthetic samples replicate public opinion" description: "Synthetic survey respondents tend to exaggerate differences between demographic subgroups and portray some responses as nearly universal." date: "2026-09-30" authors: - name: "Athena Chapekis" job_title: "Computational Social Scientist" link: "https://www.pewresearch.org/staff/athena-chapekis/" - name: "Arnold Lau" job_title: "Research Methodologist" link: "https://www.pewresearch.org/staff/arnold-lau/" - name: "Samuel Bestvater" job_title: "Associate Director, Data Labs" link: "https://www.pewresearch.org/staff/samuel-bestvater/" - name: "Sono Shah" job_title: "Former Associate Director, Research" link: "https://www.pewresearch.org/staff/sono-shah/" - name: "Andrew Mercer" job_title: "Principal Methodologist" link: "https://www.pewresearch.org/staff/andrew-mercer/" - name: "Aaron Smith" job_title: "Director, Data Labs" link: "https://www.pewresearch.org/staff/aaron-smith/" url: "https://www.pewresearch.org/data-labs/2026/09/30/how-well-synthetic-samples-replicate-public-opinion/" categories: - "Artificial Intelligence" - "Methodological Research" - "Uncategorized" --- # How well synthetic samples replicate public opinion **About this research** This Pew Research Center report examines whether AI-generated survey data can accurately duplicate the results of high-quality public opinion polls. We conducted this methodological experiment to better understand an emerging method in survey research. The Center believes that speaking to the public is essential to measuring public opinion and has no current or future plans to use AI models to generate survey results. Read more about our [AI policy](https://www.pewresearch.org/decoded/2026/07/30/how-pew-research-center-is-and-is-not-using-ai-in-our-work-2/). For this study, we had an AI model take on the role of real respondents from our [American Trends Panel](https://www.pewresearch.org/the-american-trends-panel/) (ATP) and answer the same surveys administered to human panelists. These surveys included many different question formats and asked about a wide range of topics, attitudes and behaviors. #### Why did we do this? The Center regularly studies and evaluates [advances and trends in polling research](https://www.pewresearch.org/topic/methodological-research/survey-methods/). As AI-based polling becomes [more widely used in the industry](https://www.nytimes.com/2026/04/06/opinion/ai-polling.html), we wanted to understand the data these “synthetic samples” produce and how it compares with high-quality surveys of real humans on topics of broad public interest. That question is the focus of this study; it does not address other uses of AI models in polling. #### How did we do this? We used a “digital twins” approach for this study, asking an AI model to adopt the personas of real humans who are members of the ATP. We gave the model a [wide range of information about each panelist](#_Conditioning_information), including their self-reported demographic information and their responses to questions on our [2025 political typology survey](https://www.pewresearch.org/politics/2026/06/10/beyond-red-vs-blue-the-political-typology/). We then showed the model questions from three different ATP surveys conducted in the first half of 2026 and asked it to respond to those same questions as the assigned persona. The model was given each question on the survey in order from start to finish, with exactly the same instructions and survey programming that the real humans who took it received. To make a direct comparison between our AI and human polls, we only replicated surveys for the ATP panelists from each wave who had also completed the political typology survey. As a result, figures for U.S. adults in this report may differ slightly from those previously published by the Center. We tested multiple models, but all synthetic results unless noted otherwise come from Anthropic’s Claude Opus 4.6. At the time we conducted this study, Opus 4.6 was the most recent Anthropic model available for commercial use and had the best performance of several we evaluated. Here are the [survey questions from Wave 185](https://www.pewresearch.org/wp-content/uploads/sites/20/2026/01/PP_2026-01-29_views-of-trump_questionnaire.pdf), [Wave 190](https://www.pewresearch.org/wp-content/uploads/sites/20/2026/04/pg_2026.04.28_us-role-world_questionnaire.pdf) and [Wave 192](https://www.pewresearch.org/wp-content/uploads/sites/20/2026/05/PP_2026.5.11_national-problems_questionnaire.pdf) used for this analysis, the [detailed responses](https://www.pewresearch.org/wp-content/uploads/sites/20/2026/09/pl_2026.09.30_silicon-samples_topline.pdf) from the ATP and AI surveys, and the [methodologies](https://www.pewresearch.org/data-labs/2026/09/30/methodology-silicon-samples/) for the original ATP survey waves and our AI surveys. Any evaluation of a new polling technology like AI-based surveys starts with a basic question: How well does it replicate the results of high-quality, nationally representative probability polls? Does it accurately depict the shares of Americans who hold various views? And does it work better or worse for particular demographic and behavioral subgroups? ### AI-generated survey results consistently differ from human surveys *Average absolute error (in percentage points) between human and synthetic respondents on 3 replicated survey waves* | Group | Wave 185 | Wave 190 | Wave 192 | Average across waves | | --- | --- | --- | --- | --- | | Total | 11.3 | 14.7 | 11.2 | 12.4 | | White | 11.7 | 14.1 | 11.6 | 12.5 | | Hispanic | 12.7 | 16.8 | 12.3 | 13.9 | | Black | 14.1 | 18.5 | 12.7 | 15.1 | | Asian | 12.5 | 16.1 | 11.9 | 13.5 | | 18-29 | 11.6 | 15.8 | 11.8 | 13.1 | | 30-49 | 11.6 | 14.8 | 11.6 | 12.7 | | 50-64 | 12.1 | 14.6 | 11.3 | 12.7 | | 65+ | 12.5 | 14.6 | 11.4 | 12.8 | | Rep/Lean Rep | 15.1 | 16.6 | 16.4 | 16.1 | | Dem/Lean Dem | 12.2 | 16.7 | 11.9 | 13.6 | | Some College | 11.8 | 14.9 | 11.5 | 12.7 | | H.S. graduate or less | 12.3 | 16.1 | 12.5 | 13.6 | | College graduate+ | 11.1 | 14.2 | 10.4 | 11.9 | | Men | 11.5 | 14.2 | 11.4 | 12.4 | | Women | 11.5 | 15.2 | 11.1 | 12.6 | * Estimates for Asian adults are representative of English speakers only.Source: Surveys of U.S. adults and “digital twins” synthetic analysis using Claude Opus 4.6 set to low reasoning with extended profile information and expert reflection. Surveys were conducted Jan. 20-26, 2026 (replicated March 9-12 and April 7-10), March 23-29, 2026 (replicated March 30-April 2), and April 20-26, 2026 (replicated April 27-May 1). “Can AI Stand In for Human Survey-Takers? Not Really” To evaluate these questions, we used AI respondents to replicate three survey waves previously administered to the [American Trends Panel](https://www.pewresearch.org/the-american-trends-panel/) (ATP). We then calculated the average question-level error rate for each wave, including the error rates for several demographic subgroups. **How we measured the difference between our human and AI survey-takers** To compare results from our human and AI survey-takers, we calculated the **average absolute percentage** **point error** for each survey question. Consider a question with three different response options that our human and AI survey-takers answer as follows: - 50% of humans and 70% of AI respondents choose Option A (a 20-point difference). - 30% of humans and 25% of AI respondents choose Option B (a 5-point difference). - 20% of humans and 5% of AI respondents choose Option C (a 15-point difference). The average error across those three options would be (20+5+15)/3, or 13.3 percentage points, for that question. These are the three ATP waves we replicated and some of the topics they included: - **Wave 185** (fielded January 2026): political attitudes and opinions of the Trump administration, military action in Venezuela, annexation of Greenland, conduct of U.S. Immigration and Customs Enforcement officers, and attitudes toward data centers - **Wave 190** (fielded March 2026): global politics and international relations, knowledge questions about international affairs and the U.S. Constitution - **Wave 192** (fielded April 2026): presidential approval and political attitudes, problems facing the country, military action in Iran, and sleep habits *This analysis is part of a larger evaluation of AI-generated synthetic samples in public opinion research. Read* *[a summary of the main findings](https://www.pewresearch.org/data-labs/2026/09/30/can-ai-stand-in-for-human-survey-takers-not-really/)* *and refer to the* *[methodology](https://www.pewresearch.org/data-labs/2026/09/30/methodology-silicon-samples/)* *for more details on how we conducted our synthetic poll and compared it with human polling results.* Broadly speaking, the AI survey respondents did not match the views of their human counterparts especially well. Across all three waves, the synthetic polling results differed by an average of 12 percentage points from the estimates produced by the human polls. For individual waves, absolute average error ranged from 11 to 15 percentage points. There were also clear errors for all major subgroups in the analysis. For a range of demographic and behavior groups across all three waves, none had an average error of less than 12 points. For several groups, these errors were around 15 points or more. The synthetic sample performed especially poorly when it came to replicating the responses of groups including: - Republicans and Republican leaners - Black adults - Those who have not attended college - Those who use the internet infrequently or not at all ### Synthetic poll performance on specific topics and subgroups In addition to the model’s overall poor performance replicating public opinion for different subgroups, we also found that this method produced skewed or biased results for specific questions and subgroups. #### Views of Hispanic adults toward the World Cup, Spanish classes in schools ### AI polls greatly overstate Hispanic adults’ interest in the World Cup *% who are __ to follow the World Cup* | Group | Mode | Net likely | Extremely likely | Very likely | Somewhat likely | Not too likely | Not at all likely | Net unlikely | Not sure | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Hispanic | Synthetic respondents | 97 | 4 | 52 | 41 | 2 | 0 | 2 | 0 | | Hispanic | US adults | 42 | 11 | 14 | 17 | 20 | 28 | 48 | 8 | | Non-Hispanic | Synthetic respondents | 22 | 0 | 1 | 21 | 61 | 17 | 78 | 0 | | Non-Hispanic | US adults | 25 | 5 | 7 | 13 | 20 | 51 | 71 | 4 | Note: Figures for U.S. adults may differ slightly from those previously published by the Center because they are based on a subset of respondents to the original survey. Source: Survey of U.S. adults and “digital twins” synthetic analysis using Claude Opus 4.6 set to low reasoning with extended profile information and expert reflection. Survey was conducted March 23-29, 2026 (replicated March 30-April 2).“Can AI Stand In for Human Survey-Takers? Not Really” AI models are trained on how humans think and act in large part from online content. Because of this, there has long been evidence and concern that they have a [tendency to make assumptions](https://hai.stanford.edu/news/covert-racism-ai-how-language-models-are-reinforcing-outdated-stereotypes) or rely on stereotypes about certain groups of people. We saw several instances of this behavior in our own experiment. For example, in a March 2026 ATP survey, fewer than half of real Hispanic adults (42%) said they were at least somewhat likely to follow the World Cup. In contrast, nearly every simulated Hispanic respondent in our AI poll (a full 97%) said they were likely to follow the tournament. Likewise, the synthetic sample dramatically overestimated Hispanic opinion on the importance of teaching Spanish in schools. According to the synthetic sample, 72% of Hispanic adults believe it is extremely important for schools to offer instruction in Spanish – more than double the rate among real Hispanic adults (32%). #### Partisan attitudes In addition to racial stereotyping, there were also numerous instances of our model misstating the nature and magnitude of partisan differences on issues of the day. Most notably, our AI survey often took views that are held by many Democrats or Republicans – but by no means all of them – and made them appear close to universal. For instance, **at least** **80% of Democrats and Democratic leaners in our AI survey** say: - The fact that some people in the United States have personal fortunes of more than $1 billion is a bad thing for the country (while the actual share is 45%). - They would prefer to live in an area where homes are smaller and closer to each other, but with lots of amenities nearby (actually 60%). Similarly, **more than 90% of Republicans and Republican leaners in our AI survey** say: - The police should be allowed to stop and search anyone who fits the general description of a crime suspect (actually 69%). - They have very or somewhat favorable views of Israel (actually 58%). Across numerous such examples, our AI respondents were far less politically diverse than the actual population. ### Synthetic surveys misrepresent Republican views of Israel and Democratic views of billionaires | Do you think the fact that there are some people in this country who have personal fortunes of a billion dollars or more is ... | | A good thing for the country | A bad thing for the country | Neither a good thing or a bad thing | | | --- | --- | --- | --- | --- | --- | | Synthetic respondents | Dem/Lean Dem | 1 | 86 | 14 | | | Synthetic respondents | Rep/Lean Rep | 35 | 9 | 56 | | | US adults | Dem/Lean Dem | 9 | 45 | 45 | | | US adults | Rep/Lean Rep | 31 | 14 | 54 | | | | | | | | | | Synthetic respondents | Dem/Lean Dem | 0 | 22 | 68 | 9 | | Synthetic respondents | Rep/Lean Rep | 18 | 77 | 4 | 0 | | US adults | Dem/Lean Dem | 3 | 14 | 37 | 44 | | US adults | Rep/Lean Rep | 20 | 38 | 27 | 14 | Note: Figures for U.S. adults may differ slightly from those previously published by the Center because they are based on a subset of respondents to the original survey. Source: Survey of U.S. adults and “digital twins” synthetic analysis using Claude Opus 4.6 set to low reasoning with extended profile information and expert reflection. Survey was conducted March 23-29, 2026 (replicated March 30-April 2).“Can AI Stand In for Human Survey-Takers? Not Really” --- **Next:** [Synthetic surveys and ‘timely and topical’ questions](https://www.pewresearch.org/data-labs/2026/09/30/synthetic-surveys-and-timely-and-topical-questions.md)