Methodology

This report compares synthetic polling results with benchmark surveys administered to human respondents. To develop the “silicon sample,” we first built a custom architecture for administering synthetic surveys. Then, we selected three survey waves from Pew Research Center’s American Trends Panel (ATP) – Wave 185, Wave 190 and Wave 192 – and replicated them on AI-generated “respondents” with the characteristics of the people who completed the original surveys.

The American Trends Panel survey methodology

This report is based on three surveys of Pew Research Center’s American Trends Panel (ATP), a nationally representative panel of randomly selected U.S. adults:

• Wave 185, conducted Jan. 20-25, 2026 (N=8,512). Data in this report comes from a subsample of 6,700 panelists due to the cost of running Opus 4.6. The subsample was drawn with probability proportional to their original Wave 185 weight, with all non-Hispanic Asian adults selected with certainty. The weights were then adjusted to account for further subsampling and then calibrated to align with the same population benchmarks as used in the full wave. All panelists who responded to Wave 185 also responded to the 2025 political typology survey.
• Wave 190, conducted March 23-30, 2026 (N=3,507). Data in this report comes from 3,398 panelists who responded to both Wave 190 and the 2025 political typology survey.
•  Wave 192, conducted April 20-26, 2026 (N=5,103). Data in this report comes from 4,981 panelists who responded to both Wave 192 and the 2025 political typology survey.

Only panelists who also completed the 2025 political typology survey were included in this analysis. Further methodological details about each survey wave can be found at the links above.

To learn more about the ATP, read “About the American Trends Panel.”

Silicon sample development and analysis

We constructed our AI poll using a “digital twin” approach in which each human respondent has a corresponding AI doppelganger who stands in for them in the synthetic survey.

Constructing digital twins for each respondent lets us reproduce the same kind of flexible subgroup analysis we would perform with a real survey. We can look at estimates for all respondents, and then separately look at estimates by partisanship, age, race or any other characteristic. Because every synthetic respondent has its own complete set of answers, we know these breakouts are internally consistent and will average out to match the overall figures, the same way real survey data does.

This approach also allows us to apply survey weights to the synthetic sample, which means we can target population benchmarks and adjust for coverage and nonresponse in the same way we would do for a panel of real people.

Digital twins also can introduce more variance into the data than aggregate-estimate approaches. Because each synthetic respondent answers independently, the method avoids some of the averaging that can make LLM-generated estimates appear somewhat flat or generic. It instead allows for “person-to-person” variation. Two synthetic respondents with very similar demographic and political profiles have the freedom to answer a given question differently, just as two real people might.

Finally, and perhaps most importantly for this research, the digital twin approach allows us to see how and where a model succeeds or struggles. For example, we can determine if it is consistently misrepresenting certain subgroups or failing to replicate certain types of questions, rather than simply report that a given estimate was off by some (potentially arbitrary) number of percentage points.

How we selected survey waves to benchmark

To account for the way real-time events can change people’s opinions, we wanted to replicate the survey waves as close to their original ATP field dates as possible. However, due to time and budget constraints, we only wanted to run replications for panelists who completed the survey. For this reason, survey replications were run starting the day after the real survey waves returned from the field.

There was one exception: Wave 185 was used for model configuration testing, so its replications were run for a number of weeks after the survey had fielded.

Wave selection criteria

Waves were selected for replication based on:

• Field dates. Due to the timeline of the project, we needed waves that fielded in the first half of 2026. For model testing, we needed a wave that had fielded as of February 2026. For additional replications, we needed waves that fielded between March and May 2026.
• Time constraints. To allow replications to take place without overlap, and with sufficient time to prepare the materials needed for replication, there needed to be time between waves.
• Monetary constraints. Full iterative replications are very token-intensive, meaning costs quickly add up. Based on the budget for this project, we had a limit to the number of questions we could replicate and the number of panelists we could replicate those questions for. This helped us narrow down which combination of waves would be replicated.
• Survey content. ATP waves can focus on any number of topics. We wanted to make sure there was topical diversity in the waves we replicated (i.e., not all U.S. politics). Still, the majority of questions tested ended up being on political topics.

Based on the available waves in our project window, the topics and number of questions on each, and the number of panelists set to take the surveys, we narrowed our replications down to the three waves listed above.

The LLM survey-taking pipeline

To evaluate the performance of the synthetic samples, we wanted to replicate the survey-taking process as closely as possible. To that end, the LLM “respondent” answered the entire survey one question at a time, as the corresponding human panelist might have. After each question was answered, it was added back into the prompt to allow the LLM to remember the previous questions and how it answered them. By doing this, we hoped to promote internal consistency.

Specifically, we were not interested in having the LLM generate aggregate point estimates for the entire panel or certain subgroups. When synthetic survey data is generated in this way, it has no guarantee of internal consistency and does not allow for additional demographic breaks and comparisons (the way we would analyze the results of a real survey).

While research has found that aggregate estimates are generally more accurate, this is due to the averaging effect of the LLM. We were less interested in knowing exactly how close the synthetic averages were to the real panel averages. Instead, we wanted to understand the specific ways in which trends and patterns in the synthetic data failed or succeeded in replicating human opinion.

Randomization and split forms

Each digital twin encountered the same survey programming as its human counterpart. If an ATP panelist was on Form 1 of a split-form survey, their digital twin was too. If they took the survey in Spanish, so did their digital twin.

We also programmed the same question order and response order randomization for each pair due to an interest in response-order biases – we know they are present in real survey data and wanted to know if they would be similarly present in the synthetic sample.

Profile information

To condition the LLM on each persona it was meant to emulate, we used a series of demographic variables from the ATP. In addition to basic demographic information such as age, education, race, gender and political affiliation, we included things like:

• Marital status and presence of children
• Household income
• State and county of residence
• Volunteerism
• Moved houses in the past year
• Military veteran status
• Frequency of internet use
• Religious attendance
• Disability
• Labor union membership
• 2022 and 2024 turnout and vote choice
• Political ideology (liberal versus conservative)

These variables were present in all the configurations we tested. Refer to Appendix E for the full list of standard profile variables.

The extended profile

Along with the characteristics above, we provided our AI respondents with 71 additional variables from our 2025 political typology survey.

These included broader dispositions such as whether or not most people can be trusted, how important it is to believe in God to be good and moral, and how often panelists felt unfairly judged by others because of the language they use to express themselves.

It also included more specific opinions about concrete issues such as how comfortable panelists felt with transgender athletes competing on sports teams that don’t match their sex at birth, whether the increased use of AI in daily life is generally good or bad for society, and whether or not reductions in Social Security benefits for future retirees should be considered.

Expert reflection

An “expert reflection” is a short, model-generated synthesis of a synthetic respondent’s profile, written from the perspective of a specified social-science expert. The goal in producing these reflections is to surface implications or facts about the persona that aren’t directly stated in its profile data.

Prompting an LLM with a respondent’s raw, self-reported data alone can cause the model to overlook higher-level, less obvious insights embedded in that data. To account for this, before running a synthetic survey, we prompted GPT-5.1 to adopt the persona of a domain expert and generate a set of observations about the respondent that draw out patterns that person didn’t explicitly state themselves. These reflections were generated once per respondent and appended to the prompt as part of their profile.

Some examples of the expert reflections generated:

• “Young, college-educated, high-internet urban resident in [LOCATION]: as a [AGE]-year-old Gen Z with a BA and ‘almost constantly’ online, this person is likely exposed to a wide range of digital and social networks and news sources. That exposure plus urban life in a [REGION] metro tends to produce pragmatic, information-driven attitudes and greater familiarity with diverse cultural viewpoints even if they self-label as [IDEOLOGY].”
• “Upper-income early professional who rents and lives alone: earning in the [SALARY] range at a young age suggests upward economic mobility and a stake in pro-growth, pro-market policies (taxation, regulatory environment, career flexibility). Being a renter and single also makes housing affordability, urban amenities, and career opportunities more salient than family-oriented policies.”
• “[POLITICAL AFFILIATION] who leans [PARTY] but calls themselves [IDEOLOGY] and has split tickets: the mix of [POLITICAL AFFILIATION] ID, a [PARTY] lean, self-described [IDEOLOGY], and evidence of ticket-splitting ([PARTY] congressional vote in 2022; [POLITICAL CANDIDATE] in 2024) points to a candidate-oriented, heterodox voter. They are likely skeptical of party orthodoxy, prioritize specific issues or personalities over party labels, and are open to third-party or protest votes.”
• “Cultural Catholicism with low religiosity: identifying as Roman Catholic but seldom attending services and praying, and not evangelical, suggests religion is a cultural reference point rather than a daily organizing force. This profile implies some residual influence of Catholic social norms on moral views, but lower likelihood of supporting policy positions driven by active religious commitment.”

We tested generating these reflections from the perspectives of a political scientist, a behavioral economist and a sociologist but did not notice substantial differences in their results. Ultimately, we used the “expert political scientist” reflections for this project.

Prompting

To generate responses, the model was given a system prompt establishing its task: to role-play as a specific survey respondent, using the profile information described above to inform its answers. The prompt instructed the model to answer as that person would, taking into account any questions and answers the synthetic respondent had already provided earlier in the same survey so that its answers remained consistent with its own prior responses as the questionnaire progressed.

The prompt also included several instructions meant to push the model toward more realistic, humanlike survey-taking behavior. It reminded the model that it was playing a role, meaning it shouldn’t answer with more knowledge, precision or consistency than a real person would have, and that it was acceptable for its “knowledge” to be incomplete or mistaken in ways that fit the respondent’s profile. It permitted the model to decline to answer a question when that would be realistic (for instance, out of disinterest or if a question topic felt too sensitive).

Finally, the prompt asked the model to take into account how real survey respondents are known to behave (including common cognitive biases such as question-wording effects, response-order effects and context effects) and to allow its answers to be shaped by these same biases where appropriate, rather than answering as an idealized, bias-free respondent would.

Refer to Appendix A for the full prompt text.

Determining the best configuration

We tested three different dimensions along which the model configuration could differ: reasoning level, the amount of profile information used to create the “digital twin” and the choice of model itself. Due to time and budget constraints, we could not test every combination of these dimensions. Instead, we opted for a bracketed (tournament-style) approach in which each dimension was tested one by one, with the winner of each round advancing to the next.

Initial testing (until model evaluation) was done using GPT-5.1, which was the most recently released OpenAI flagship model available at the time. We used GPT for testing, as it is generally considered an industry “baseline” model.

Reasoning level

We started by comparing low, medium and high model reasoning levels. Increasing the amount of reasoning greatly slows down the LLM’s survey-taking speed, so this choice would have the biggest impact on our timeline. Each configuration was built on a simple, random subsample of 237 respondents from Wave 185 and the standard profile variables.⁴

The configurations were compared using simple match rates (whether the category selected by the LLM matched the category selected by the ATP panelist) in two ways:

• Person-level match rates. For each person, the number of questions correctly matched by the LLM over the total number of questions asked.
• Question-level match rates. For each question, the number of people whose answers were correctly matched by the LLM over the total number of people asked.

These match rates were not expected to be anywhere close to 100%, so they were solely compared with one another.

The average person-level match rate was approximately 55% under low, medium and high reasoning levels. This match rate varied by education, age, race, gender and party identification. However, person-level match rates across reasoning levels did not differ by more than 1 percentage point within any of these subgroups.

Similarly, the average question-level match rate was approximately 54%-55% across all three reasoning levels. There was also no configuration that performed noticeably better or worse at the question level: High reasoning achieved the best question-level match rate for 43 questions, medium reasoning for 39 and low reasoning for the remaining 37.

There was no evidence to justify the additional time and financial costs of higher reasoning levels. Thus, we selected low reasoning for all subsequent replications.

Conditioning information

Using GPT-5.1 with low reasoning, we then compared a configuration with only standard profile information to one with the additional variables on political attitudes and values (i.e., the extended profile), as well as to an extended profile that incorporated “expert reflection.” Each configuration was run on all respondents to Wave 185 (N=8,512).⁵

Since here we replicated a full survey, we used a more flexible comparison metric than match rate. We calculated the average absolute percentage point error between the overall weighted estimate from each configuration and the ATP data. Specifically, we first average the absolute errors across each response option for each question, then average those averages to get a single number – the survey average absolute error – that summarized how well each synthetic sample performed. The further away from 0, the more inaccurate the synthetic estimate.

Including only standard profile information resulted in a survey average absolute error of 16 percentage points. The addition of extended profile variables made a marked improvement, shrinking the survey average absolute error to 13.7 points. Adding the expert reflection further improved it to 13.1 points.

Because these are averaged across more than 100 questions, even small differences are indicative of real gains in performance. Therefore, we selected the configuration with the extended profile variables and an expert reflection step.

Model selection

Finally, keeping reasoning level and the profile information constant, we tested three different off-the-shelf closed models: GPT-5.1 and GPT-5 nano from OpenAI, and Claude Opus 4.6 from Anthropic.

As Opus 4.6 was much more expensive to run than OpenAI’s models, we subsampled 6,700 panelists who completed Wave 185 rather than using the entire sample. This subsample was drawn in two replicates. The first replicate (N=3,200) was made up of all non-Hispanic Asian panelists and a random one-third of the remainder.⁶ The second replicate (N=3,500) was sampled from cases not in the first replicate with probability proportional to their final Wave 185 survey weight. The survey weight for the resulting subsample of 6,700 panelists was then adjusted for each person’s probability of being subsampled.

While the configurations using GPT-5.1 and GPT-5 nano were run using all 8,512 panelists who completed Wave 185, all comparisons between them and Opus 4.6 are filtered only to the N=6,700 subsample.

GPT-5.1

This was our baseline model for testing. GPT-5.1 is a mid-priced option – not the cheapest available but far less expensive than top-tier, flagship models. At the time of this analysis, it was OpenAI’s most recently released model. As of 2026, GPT is generally considered industry standard for the kind of LLM task involved in this research.

GPT-5.1 with the extended profile variables and expert reflection step achieved a survey average absolute error of 13.3 percentage points.[7 This is the same configuration used when comparing different amounts of profile information, but the survey average absolute error metric differs slightly due to the subsampling.]

GPT-5 nano

GPT-5 nano is a comparatively cheap, lightweight model and runs faster than a model like GPT-5.1. However, this speed and low cost come with a real tradeoff in quality: Independent benchmarks put GPT-5 nano’s intelligence index score on the lower end across tested models. This gap showed up clearly in our results – GPT-5 nano performed noticeably worse than our baseline, confirming that model version and size matter a great deal for this kind of task.

GPT-5 nano achieved a survey average absolute error of 17.4 percentage points, which would be less accurate than GPT-5.1 even with only the standard profile variables.

Opus 4.6

We also wanted to test a state-of-the-art reasoning model outside the OpenAI model family. Claude Opus 4.6 is priced roughly five times higher than GPT-5.1, but independent benchmarks scored it among the highest intelligence models as of February 2026. In our tests, Opus 4.6 performed consistently better than GPT-5.1 across the board.

Opus 4.6 achieved a survey average absolute error of 11.4 percentage points, thus outperforming both OpenAI models tested under the same conditions.

Based on the combination of parameters with the best performance, our final configuration used Opus 4.6 with low reasoning, extended profile information and expert reflections.