Polling organizations devote a great deal of attention to the wording of survey questions, but they need to be just as concerned about how questions are ordered. The context in which a specific question is asked, particularly what directly precedes a question in a telephone interview, often has an impact on the way people respond.
In our most recent survey on the budget deficit, we asked three separate, but similarly worded questions of 1,024 respondents: Whether it is more important to take steps to reduce the budget deficit or: 1) to keep spending for programs that help the poor and needy at current levels; 2) to keep military spending at current levels; 3) to keep Social Security and Medicare benefits as they are. The order in which the three were asked was randomized, meaning that different respondents were asked the questions in a different sequence.
The survey found that among all respondents, 69% prioritized maintaining current Social Security and Medicare benefits over reducing the budget deficit; in March 2013, somewhat fewer (57%) said this.
On the surface, this appears to be a substantial change in opinion. But in the past, the question comparing Social Security and Medicare benefits with deficit reduction was asked alone. In the current survey, because we wanted to address more than one area of spending relative to the deficit, it was randomized along with questions on military spending and programs for the poor, meaning that for about two-thirds of respondents the question about Social Security followed questions about other deficit tradeoffs.
To determine how context might have affected the results, we reviewed the data to see whether the question order made any difference in the results. Among respondents who were asked the Social Security/Medicare question first – before the questions on military spending and programs to aid the poor – 63% said it was more important to maintain benefits as they are, while 30% placed greater priority on the deficit. Among those who were asked the same question later, a higher percentage (72%) said it was more important to maintain benefits. We observed a similar pattern for the questions addressing military spending and programs to aid the poor.
This particular type of question ordering effect is known as a “contrast effect,” meaning that the order in which the questions were asked produced different results. It raised doubts about whether there was a change in opinion on this question since we last asked the Social Security/Medicare question in March. For this reason, we did not report on the trend on this question in the report and noted the different context for the question in the survey topline.
While the order in which respondents heard the questions mattered, the overall results provide a good picture of how the public views different tradeoffs with respect to deficit reduction. Protecting Social Security and Medicare ranked higher in people’s minds irrespective of whether they heard it first, second, or third. And the randomization of the questions ensured that no “order effects” produced an undue bias against any one issue.
Not all questions that are randomized show this type of effect. In this case, the “contrast effect” may be related to the public’s views about the importance of taking action on the deficit. Prior Pew Research Center polls have found that deficit reduction rates as one of the public’s top policy priorities for the president and Congress. For questions coming second or third, rather than first, the context of budget deficit vs. spending had already been established. Respondents had already had the opportunity to potentially express support for reducing the deficit, and may have considered the attractiveness of one area of domestic spending relative to at least one other.
