May 8, 2014

Q/A: How Pew Research tracks public opinion in countries stricken by violence and unrest

BellThe Pew Research Center today released findings from a new survey in Ukraine, conducted as that country grapples with separatist movements, armed conflict and general uncertainty about the nation’s political future. Pew Research is polling in more than 40 countries this year. Fact Tank’s Drew DeSilver sat down with James Bell, Pew Research’s director of international survey research, to discuss how the center designs and implements its surveys in places like Ukraine.

Q: Many people following the news in Ukraine are likely wondering, how was Pew Research able to successfully field a survey under present conditions? 

A: In Ukraine, as in many other countries around the globe, our surveys are conducted face-to-face, rather than by telephone. This means that interviewers are assigned the task of walking through neighborhoods and knocking on doors to randomly select survey participants. Being out on the street can be risky at times, especially in a restive country like Ukraine. I want to stress that Pew Research always puts interviewer safety first. Before launching a survey in a foreign country, we discuss security and other concerns with our lead-contractor (in this case, our long-standing partner, Princeton Survey Research Associates International), and the local polling firm that we hire to administer the interviews. If we judge conditions in a country to be too dangerous, we will either delay or cancel the project. We never require interviewers to knowingly risk their personal safety.

Q: Can you provide some examples of countries where Pew Research has either delayed or reconsidered a project due to security fears?

A: In spring 2013, our survey in Venezuela was delayed for several weeks due to political unrest in the country. Interviewers were unable to safely travel to their assigned neighborhoods, and completed questionnaires could not be securely transported from the provinces to the central office where data were entered into a computer. Fortunately, conditions sufficiently improved so that the local polling firm we worked with could complete the necessary interviews and process the data for delivery to us.

Sometimes the duration or scale of a safety threat requires us to go back to the drawing board and assign new locations to interviewers, so they can complete the survey. An example is Afghanistan, where armed confrontations between the army and Taliban in 2011 required us to substitute a number of randomly selected villages and towns with safer alternates. This was done in a way that preserved the representativeness of the survey. In other countries, such as Pakistan, we know going into a project that we are unlikely to be able to safely interview people in all parts of the country. If you check our methods statements for Pakistan, you will see that we consistently exclude the Federally Administered Tribal Areas and certain other regions due to security concerns. Even with these omissions in coverage, our survey in Pakistan accurately represents the opinions of the overwhelming majority of Pakistanis.

Q: Returning to Ukraine, did Pew Research have to exclude Crimea or other parts of the country due to unrest and violence?

A: No, we did not. In large part, this was due to the fact that our local partner in Ukraine has a network of experienced interviewers throughout the country. Under normal conditions, at least some of these interviewers would travel between several locations to complete interviews. However, given the current unrest and reports of checkpoints on certain roads manned by either government or opposition forces, we decided that interviewers would only be assigned to randomly selected locations near interviewers’ homes.

In addition to keeping the fieldwork team safe, this approach meant that when a respondent opened their door to an interviewer, they encountered someone who spoke the same language, had a similar accent and generally felt like a neighbor. Interviewers, of course, clearly identified themselves as employees of a professional research firm. But I think the sense of shared local identity between respondent and interviewer helped to reduce possible suspicions of the survey and its motives. Interviewers were aided in this regard by having the option of administering the survey in either Ukrainian or Russian, based on the respondent’s preference.

Q: So, Pew Research faced no interference in conducting its survey in Ukraine?

A: We encountered one instance where a driver on his way to deliver completed questionnaires to the central office was stopped at a roadblock. Unidentified men seized a packet of questionnaires and burnt it on the spot. The driver was not detained and left the scene unharmed. This occurred in the early days of fieldwork, and the lost questionnaires were replaced by interviewers returning to the original neighborhoods and re-administering the survey. The incident certainly put us and the local firm on alert. We immediately emphasized to our local partner that they should cease fieldwork if they sensed a risk to their employees. Given the randomness of the roadblock, our partner decided to cautiously proceed with interviewing. In the end, it proved to be the only interference during fieldwork.

Q: Wouldn’t it be easier, not to mention safer, to interview people in Ukraine and other restive countries by phone?

A: We regularly conduct surveys by telephone in countries like the United Kingdom, France, Germany, Spain and Japan. These countries are characterized by widespread ownership of both landline and mobile phones. They are also countries where we feel confident that we can sample both landline and mobile phone users at a reasonable cost, and without missing key segments of the population. In many countries, including Ukraine, landline phones were never as ubiquitous as in western countries and the advent of mobile phones has not yet translated into what we would consider both an economical and rigorous alternative to face-to-face surveys.

Q: I noticed the Ukraine survey included an oversample of the country’s eastern region as well as Crimea. How do you make decisions about when to oversample?

A: Whether in Ukraine or elsewhere, it is sometimes important to have survey data that speaks reliably about members of subgroups. In national surveys, a standard way of achieving this is by selecting more people from such subgroups than would typically be done if everyone in the sample had an equal chance of being selected, a technique known in opinion research as an ‘oversample.’ Our oversample in eastern Ukraine and Crimea, for instance, gives people in these regions a greater chance of being randomly selected by boosting the number of interviews assigned to these areas of the country.

To make this a bit more concrete, people living in eastern Ukraine and Crimea make up roughly 57% of Ukraine’s population, but 72% of all the respondents we interviewed for this survey. The benefit of the oversample is that when we analyze the attitudes of people within eastern Ukraine or Crimea, we have a larger number of respondents that we can analyze, and are able to identify important subgroup differences, such as that between people in this region who speak Russian only, as opposed to Ukrainian only. Our report highlights some of the dramatic ways in which these groups differ in their opinion of the current government in Kyiv and their views of Russia. However, when we talk about the total share of Ukrainians who hold a certain opinion or report a behavior, people in eastern Ukraine and Crimea are weighted “down” to their actual share of the population, 57%.

Topics: Polling, Research Methodology, Wars and International Conflicts

  1. Photo of James Bell

    is Director of International Survey Research at the Pew Research Center.

Leave a Comment

Or

All comments must follow the Pew Research comment policy and will be moderated before posting.

8 Comments

  1. Sinnathamby7 months ago

    Today’s democracy depend on public opinion. Armed thrust against any nation’s opinion will soon suffer a defeate on the face of world opinion.

    Reply
  2. Terry7 months ago

    I would like to know the exact question for the PEW survey since it could skew the poll. For example if the question was: Do you prefer to remain part of Ukraine?; a large number might say yes, but if those same people were ask the following question: Do prefer to remain in Ukraine under the current government in Kiev? A suspect a lot those yes’s become no’s. I believe this explains the big difference between the PEW results and the referendum results.

    The other factor to consider: How recent is the PEW survey. If the PEW survey is more than a few weeks old, it might currently be inaccurate. I suspect opinions are quickly changing in Russian areas of Ukraine. Resentment of the new government in those areas is clearly rising. We in the west need to set politics aside and seek honest answers regarding the reality in Ukraine.

    Reply
    1. Dan Joyner7 months ago

      Great Questions.

      Everything you want to know can be found here:

      pewglobal.org/2014/05/08/despite…

      Happy reading!

      Reply
  3. Mark7 months ago

    Very interesting. I have discussed the Crimea findings here: humanrightsinvestigations.org/20…
    I agree with calripson that it would be interesting to get a break-down of attitudes in Donetsk and Luhansk.

    Reply
    1. Daniel Joyner7 months ago

      Everything you want to know can be found here: pewglobal.org/2014/05/08/despite…

      Reply
  4. calripson7 months ago

    There is an obvious problem with the polling as demonstrated by the percent of Crimeans stating they support the right to secede and the actual percentage who actually voted to secede. Even factoring in the lack of participation from Tartars in Crimea, the percent of the population supporting secession was around 82% which matches very well with earlier votes in the 1990s adjusted for shifting demographics.

    Also interesting is the definition of “Eastern Ukraine” which includes regions such as Poltava, Sumy, Kirovhorad, and to a lesser extent Dnipropetrovsk not considered pro-Russian. The real question is what is the actual support for leaving Kiev in Donetsk and Luhansk. It would be interesting to see these regions broken out separately.

    Also, given the attitude of the government towards separatist (deemed terrorists) and the bias coverage of Ukrainian television it is highly unlikely that people outside of Donbass would feel comfortable giving a “pro-separatist” answer. The Right Sector has in fact been asking people similar questions and if they receive the wrong answer they beat and sometimes kill the offender.

    My sense of the actual level of support for greater autonomy in Donetsk is around 70% and in Lugansk in the 60% range. Majorities but not overwhelming. This is still significant as Donetsk has a population of 4.3 million and Luhansk (Lugansk in Russian with a hard g sound) around 2.5 million.

    Reply
  5. Peter Nockolds7 months ago

    ‘Eastern Ukraine’ here is a large area and we might expect support for separatism to increase as we move eastwards. I hope that commentators will not rely too much on these results, as they stand, in assessing the results of the unoffical referenda in Donetsk and Luhansk on Sunday.

    Reply
    1. Orwell7 months ago

      Agree, the report included central parts and South into the East Ukraine. Unfortunately, BBC has already used this report to highlight the ‘fact’ that ‘Eastern’ Ukraine 70% for being united.
      Another bias in the report is a very small sample of locations. Yet, they claim 3.3% error.
      The ‘full report’ does not show data per location. Why hide it?
      It does not look a useful and accurate report, although one can still see the problem in the country.

      Reply