The research literature on using mobile apps for survey research is sparse. The majority of research has not been published but was rather publicly presented at conferences such as the American Association for Public Opinion Research’s annual meeting. The bulk of the studies were conducted by the Nielsen Company, primarily in regard to their TV viewing diary app.
While these studies focus on the use of apps and mobile devices more generally, their results are limited for the purposes of the Pew Research Center. Many focus on game and social mechanics, which is less than ideal number of reasons. The burden in terms of both cost and time to program them is too high (Lai, J., M. Link, L. Vanno (2012)). They appeal most to certain subsets of respondents, such as young adults, technology savvy respondents or minorities (Lai, J., M. Link, K. Bristol, S. Duan (2014); Lai, J., K. Bristol, S. Duan, M. Link (2013); Link, M., J. Lai, K. Bristol, S. Duan (2013); Lai, J., M. Link, L. Vanno (2012)). Finally, there is some concern that “gamification” could undermine efforts to convey that the seriousness of a political or social science research endeavor.
In addition, many of the studies that have been made public thus far were not randomized controlled experiments, limiting the ability to apply their findings to other research. For example, in some cases users were able to self-select whether they used the mobile app, a web browser (“web app”) or both (Lai, J., M. Link, K. Bristol, S. Duan (2014)) to take the surveys. In others the focus was on the device respondents chose to use to complete the survey (computer vs. mobile device) meaning it was not possible to know whether mobile device users were then using the mobile app or the web browser (Link, M., J. Lai, K. Bristol, S. Duan (2013); Link, M., S. Duan, K. Bristol, J. Lai (2014)). In instances where there was a randomized controlled experiment, it was testing the effects of introducing game and social mechanics (Vanno, L., J. Lai, J. Scagnelli, M. Link (2012); Link, M., J. Lai, and L. Vanno (2012)).
Finally, in many of the presentations the outcomes measured were based on self-reported attitudes and behaviors from a follow up survey, rather than actual outcomes of the experiment itself such as response rates or data quality (Lai, J., M. Link, K. Bristol, S. Duan (2014); Lai, J., K. Bristol, S. Duan, M. Link (2013); Lai, J., M. Link, L. Vanno (2012); Link, M., S. Duan, K. Bristol, J. Lai (2014)).
One particularly relevant Nielsen non-TV diary study (Scagnelli, J., J. Bailey, M. Link, H. Moakowska, and K. Benezra (2012)) tested event-contingent experience sampling in the context of a food purchase study over a month long field period. However, its goal was to test feasibility rather than testing and comparing different approaches. The study used a non-probability sample of 268 Millennials in southern California, and provided respondents with smartphones pre-loaded with an app in order to have the capacity to capture GPS location, use barcode scanning and take pictures. The study found that there was some change in respondent behavior over the course of the field period, in terms of participants reporting that they became more aware of purchases and opted for healthier and cheaper options. A second relevant finding and an advance in terms of the possibility of verification: the GPS capability on the phones showed that most respondents completed their surveys away from the point of sale, contrary to instructions.
Another Nielsen non-TV diary study (Scagnelli, J., K. Bristol (2014)) also used event-contingent experience sampling. The goal of the study was to measure feasibility of collecting data with this method, and employed a nationally representative pilot study in South Africa involving barcode scanning of household purchases.
In summary, while several interesting studies have used smartphone apps, they had limited relevance for the types of studies conducted by the Pew Research Center. There is a gap in the literature for studies that use a probability sample with a randomized controlled experiment unrelated to TV viewing or purchasing behavior.
