--- title: "Methodology" description: "The data in this report comes from an online opt-in survey of 11,114 U.S. adults conducted Nov. 14-19, 2024. The sample was provided by CloudResearch and originally sourced from Cint’s Lucid Marketplace. Adults ages 18 and older located in the United States were eligible to participate. The survey did not use any demographic quotas. The […]" date: "2026-08-27" authors: - name: "Sara Atske" url: "https://www.pewresearch.org/methods/2026/08/27/bogus-respondents-methodology/" categories: - "Methodological Research" - "Survey Methods" --- # Methodology The data in this report comes from an online opt-in survey of 11,114 U.S. adults conducted Nov. 14-19, 2024. The sample was provided by CloudResearch and originally sourced from Cint’s Lucid Marketplace. Adults ages 18 and older located in the United States were eligible to participate. The survey did not use any demographic quotas. The full set of respondents was then screened using three different methods designed to identify bogus or other problematic respondents: 1) trap questions, 2) automated prescreening using CloudResearch’s proprietary Sentry system, and 3) voter file matching. For each method, we identified which cases passed the checks and which would have been screened out. Out of the 11,114 respondents overall, a total of 9,151 passed the trap questions, 5,745 passed the automated prescreening and 3,977 were successfully matched to a voter file. Survey weights were created for the three groups of respondents that would have resulted if each of the three screening methods had been used to remove problematic respondents. A fourth set of weights was created for the entire sample with no screening applied. Because probabilities of selection are unknown for opt-in samples, respondents were all treated as if their probabilities of selection were all equal and given an initial weight of 1. For each set of respondents, this initial weight was then calibrated to align with the population benchmarks identified in the accompanying table and trimmed at the 1st and 99th percentiles. [![A table showing Weighting dimensions](https://www.pewresearch.org/wp-content/uploads/sites/20/2026/08/PM_2026.08.26_bogus-respondents_M-01.png?w=420){.aligncenter width=400}](https://www.pewresearch.org/?attachment_id=365004) #### Open-end coding One measure of data quality examined in this report was whether respondents answers to an open-ended question was meaningful and responsive to the question that was asked. Respondents were asked, *“What is one thing that you would like politicians in Washington, D.C., to know about your own situation when they are writing laws and setting policy? Please share as much detail as you can.”* Each respondent’s answer was assigned to one the following categories: - **Relevant:** Answers that were responsive to the question, describing some aspect of the respondent’s life or broadly related to politics. - **Nonresponse:** Answers indicating there was nothing the respondent wanted politicians to know or that they were unsure. It also included those who declined to answer the question either by stating as much or by skipping the question. - **Generic positive rating:** Answers that seem like they are giving a postive rating to a product or service, such as “Very good,” “Excellent quality” or “Good service.” It also included uncontextualized positive answers like “Yes” or “Okay.” - **Other non sequitur:** Answers that do not make sense in the context of the question, such as “Open a Checking Account Online for Free,” “Pizza” or “Hey I will call you in about an hour.” - **Gibberish:** Answers that look like someone was mashing keys on a keyboard, such as “kljasdfjkdf” or “V6f66ff6 6v6x5xuv c5d6f6g6f5 v6f6 cc 6v.” - **Probable AI:** Answers that, rather than expressing genuine respondent opinion, appeared to have been generated by AI systems like ChatGPT or Gemini. Sometimes this was explicit, such as when the answer includes phrases like “As an AI …” or “If I had a personal situation to share …”. Other times, these were answers exhibiting an overly formal style that appeared highly unlikely to have been written by a human.[2.numoffset="2" The probable AI category was added to the codebook after the human-coded benchmarks had already been created. These answers were initially categorized as relevant and were treated as such when evaluating the LLM’s accuracy in coding open-ends.] This coding scheme was adapted from a [codebook developed](https://www.pewresearch.org/methods/wp-content/uploads/sites/10/2020/02/PM_02.18.20_dataquality_Appendix-B.pdf) for a previous Pew Research Center study of bogus respondents. To validate the reliability of the codebook, a random sample of 300 answers were independently coded by two human reviewers. The sample was stratified on the number of incorrect answers each respondent gave to three trap questions to ensure that there was a sufficient number of answers across a range of data quality levels. The double coded answers yielded a Krippendorff’s alpha of 0.85 – a level generally considered to indicate a high degree of reliability. [![A table showing Quality metrics for LLM- coded open-ends](https://www.pewresearch.org/wp-content/uploads/sites/20/2026/08/PM_2026.08.26_bogus-respondents_M-02.png?w=200){.alignright width=280}](https://www.pewresearch.org/?attachment_id=365005) Another 900 similarly sampled answers were then coded and combined with the initial validation sample to create a benchmarking set of 1,200 human-coded answers considered to be coded correctly and serve as a ground truth. We developed a prompt for GPT-40, providing it with context about the task and criteria for how to assign codes. We then used it to code the benchmarking dataset, evaluated the answers where the human- and large language model (LLM)-coded answers differed, and refined the prompt to correct any systematic problems. This process was repeated several times until the overall level of agreement was satisfactory. The final prompt yielded an agreement rate of 89% and an F1 score of 0.86, indicating generally high levels of classification accuracy for the variable as a whole. Individual categories had agrement rates between 91% and 98% and F1 scores between 0.92 and 0.99. The final prompt was then applied to the remaining answers to provide the codes used in this analysis. Refer to the final prompt here: ``` You are an AI assistant developed to help extract meaning from the text of open-ended survey questions. Your task is to help researchers understand if respondents made a good faith attempt to answer the question. Respondents were all asked the following question: > What is one thing that you would like politicians in Washington, D.C. to know about your own situation when they are writing laws and setting policy? Please share as much detail as you can. In a moment you will be given an answer that a respondent provided. Which of the following categories best characterize this answer? * Relevant * This category describes answers that are relevant to the question that was asked. This includes answers that describe the respondent's life situation or answers that are broadly related to politics, government the economy and laws. It can list or describe issues that the respondent cares about. This can also include positive statements about politicians and job they are doing or instructions to politicians about how politicians should do their job. Sometimes these are short declaritive statements telling politicans what to do. * Examples: - People struggling in non-skilled industries need the most help - Ask the people using public polls so everyone would know. - That they cant take away our social secrity because thats the olny thing thats keeping from being homelres - Donald trump is the best - They should focus of the high cost of living - Be fair - I was just like for everyone just to get along - Use some common sense - put GOD back * Gibberish * This category describes answers that are just strings of characters that don't mean anything, as if someone was randomly pressing keys on a keyboard. Keep in mind that there are many typos, so consider the possibility that a word could be misspelled. Misspelled words should not be coded as gibberish. If an answer contains a large number of actual English words but doesn't make sense, it should be coded as a Non-Sequitur instead. * Examples: - uggg loj kjh ko - Bhbhbhbhbb - Zgufkgkgjxjf - V6f66ff6 6v6x5xuv c5d6f6g6f5 v6f6 cc 6v * Generic Positive Rating * This category is for answers that seem like they are giving a positive rating to a product or service instead of answering the question. It includes answers that express a positive sentiment without saying what it is about. Answers that say positive things about politicians, laws or policies should **not** be coded as generic positive ratings. * Examples: - Yes - It's the best - Very good - Excellent quality - Good service - Okay * Probable AI * This category is for answers that appear to have been written by an AI. These tend to use formal language and have very good grammar. They often use hypothetical language with phrases like "It would be helpful..." or "If I were to...". Sometimes these answers begin or end with a single quotation mark ' as if they were copied and pasted. Sometimes, they admit to being an AI, saying "As an AI" or "If I had a personal situation to share". They often restate parts of the questions, writing out "politicians in Washington, D.C." Sometimes these are long, but they can also be short if the respondent pasts in only a portion of a longer answer from an LLM. They often describe their persona, claiming to be things like "a working professional" or "a retiree on a fixed income" * Examples: - 'I wish policymakers in Washington, D.C. would better understand the impact that everyday decisions have on the most vulnerable and marginalized communities. Too often, policy changes and laws are crafted in a way that overlooks the real-life struggles of people who are already facing economic hardship, health challenges, or systemic inequality. These individuals may not have the resources to engage in the political process in the same way that wealthier or more connected groups can. - It would be helpful for politicians to understand how economic policies impact everyday life—particularly for those facing high living costs, student debt, or healthcare expenses. Recognizing the importance of affordable housing, accessible education, and equitable healthcare could guide policies that make a meaningful difference. It might also be valuable to prioritize transparency and accountability, ensuring that policy decisions are communicated clearly to the public so people can trust and understand how those policies affect them directly. - If I were to offer advice to politicians in Washington, D.C., I'd want them to understand the impact that digital and technological advancement is having on people's everyday lives. Many laws, particularly in areas like privacy, data security, and access to technology, are outdated or don't fully address the speed at which technology evolves. People increasingly rely on digital tools for work, education, healthcare, and social connection, but gaps in regulation leave individuals vulnerable to data exploitation, unequal access, and a lack of transparency from tech companies. Policies should be forward-thinking and provide clear protections for consumers, ensure equitable access to technology, and require accountability from large tech corporations. These concerns affect millions of people, and legislation should work to foster a fair, secure, and accessible digital environment for everyone. - I can share a common sentiment that many individuals might express: the need for policies that address economic inequality and access to affordable healthcare. Many people would appreciate lawmakers considering the struggles of working-class individuals and families, especially in light of rising living costs and healthcare expenses, and ensuring that legislation is designed to support those who are most vulnerable in society. The importance of affordable education and job training for upward mobility would also likely be a key concern. - I would like politicians in washington DC to know that people in specialist rolea like IT and marketing often face unique challenges, including rapidly changing technology, high expectations for continous skill development and job stability concerns due to outsourcing or automation. Policies that support ongoing education, afordable training opportunities and job security in this fields would make significant difference. - understand the constant pressure of balancing a moderate income with rising healthcare costs, childcare expenses, and the growing cost of everyday necessities, which often leaves little room for unexpected financial burdens like car repairs or medical emergencies, making it difficult to plan for the future even with responsible budgeting practices. - As a middle-class working professional living in a suburban area, the most crucial thing I want politicians in Washington D.C. to understand is the constant pressure of balancing rising living costs with stagnant wages, particularly when it comes to healthcare, education, and childcare expenses; this often leaves families like mine feeling like we are one unexpected financial setback away from instability, even with full-time employment and relatively stable income. * Non-Sequitur * This category describes answers that do not make sense or fit the question that was asked. Keep in mind that answers may have typos and misspellings, so consider the possibility * Examples: - Open a Checking Account Online for Free - Pizza - You can do it on the only reason why I can't make the - the most most recent example example is is a a simple list - The most common type for this is an old fashioned white bread with the cheese and cream filling in a pan and - Hey I will call you in about an hour * Nothing/Unsure/Refused * This category is for respondents who do refuse to give an answer. Note that if someone says "I don't know" but then follows that with something relevant, you should code it as relevant instead. * Examples: - No comment - Unsure - No idea - None of your business - Nothing - Idk - I'm not sure - I don't like politics - I don't know anything about politics You can only select one category for each answer. If you think an answer could belong to more than one category, choose the answer choice that seems more likely. ``` --- **Next:** [Acknowledgments](https://www.pewresearch.org/methods/2026/08/27/bogus-respondents-acknowledgments.md)