This signal-contingent experience sampling method study was conducted using the Pew Research Center’s American Trends Panel, a probability-based, nationally representative panel of US adults. Panelists were recruited from the 2014 Political Polarization and Typology Survey, a large (n=10,013) national landline and cell phone random digit dial (RDD) survey conducted January 23rd to March 16th, 2014 in English and Spanish. At the end of that survey, respondents were invited to join the panel. In 2014, panelists took surveys monthly, primarily by web with a small portion taking the survey by mail and sometimes phone.
This study began during the October 2014 monthly wave of the panel, conducted October 3rd to 27th, 2014 (n=3,181). All panelists were asked what type of cell phone they had. Those with an eligible smartphone (iPhone, Android or Blackberry) were randomly assigned to the web or the app treatment and asked to participate in a follow up study using the respective data collection mode. Sixty percent of the eligible panelists were assigned to the app treatment and 40% to the web treatment. Panelists with a smartphone that was incompatible with the survey app were assigned to the web group but are excluded from the experimental analysis (they are called the “non-experimental web group”). Panelists who were assigned to the app group but refused were invited to take the follow up survey via the web (“APP2WEB group”). The reason for this design was so that combined respondents from all groups — the app group, the experimental web group, the non-experimental web group and the APP2WEB group – would be representative of all smartphone users, while the first two groups could be compared for experimental purposes. The results presented here focus mainly on the app and experimental web groups. ConfirmIT software was used for the web surveys and the ConfirmIT SODA app was used for the app surveys.
The survey questions and text for the invitation to participate in the signal-contingent experience sampling method follow up study can be found in Appendix A. The invitation explained that respondents would be alerted twice a day for seven days when it was time to complete a two minute survey for a total of 14 two-minute surveys. The app group was told they would need to download an app to take these surveys while the web group was told they could use their smartphone, tablet, laptop or desktop. The invitations to these surveys would come via text message, email and for the app group, via push notification. Respondents would be given $5 for agreeing to take part in the follow up surveys, $1 for each completed survey and an additional $5 for completing 10 or more of the 14 follow up surveys, potentially totaling $24. The invitation to participate also contained a disclaimer to the app group that the app would not collect any additional data from their smartphone, but that it was a third-party app and thus could have some risk associated with it. Finally, the invitation explained that the purpose of the study was to understand how and why people use their smartphones and how it makes them feel.
All who agreed to participate in the study were then asked for consent to receive text messages with survey invitations, as the surveys needed to be completed within two hours of receiving the invitation. For those who consented, cell phone numbers were confirmed or obtained. Finally, panelists were told the alerts themselves would come at a different time each day, adjusted for the time zone, so the final question confirmed or obtained the panelist’s time zone for the period in question (November 10th to November 16th, 2014).
All panelists who agreed to participate in the follow up study were sent an advance letter and a $5 incentive if a mailing address was on file. All panelists were also emailed the same advance message. For those without a mailing address on file, the email contained a $5 Amazon gift card. Those in the app group also received instructions for downloading the app in the mailed letter and a link to do so in the email
Surveys were conducted at the days and times (in the respondents’ reported local time) shown in the accompanying table. The surveys were open for two hours each, as the content was time sensitive. The web group received a text message with a link to take the survey, while the app group received a text message telling them to go to the app; this message included a link to download the app if they had not already done so. Both groups received emails with this information as well. The app group received a push notification that took them directly to the survey in the app.
The survey itself asked respondents about the activities they had engaged in on their cell phone in the past hour, aside from answering the survey. If the respondent had used their cell phone in the past hour, they were asked where they were when they used it, what they used their phone to help them do (e.g., look for directions, settle an argument or avoid being bored), and what feelings they experienced regarding their cell phone during that time. If they had not used their phone in the last hour they were asked when they most recently used it, why they did not use it in the last hour and whether this was typical usage for them. Appendix B includes the full the questionnaire. For full results of this survey please see U.S. Smartphone Use in 2015.
