Methodology

Sample

[2]

The abridged NCI sample included six newspapers-three “tier 1” national newspapers with the highest print circulation, and three high-circulation “tier 2” major metropolitan newspapers that were a part of PEJ’s NCI coding rotation.

Each of the three domestic cable outlets entirely devoted to news were included.

In addition, one network news division was included. In order not to over-sample NBC News, given the presence of MSNBC in the sample, researchers selected the news division that, next to NBC, had the largest viewing audience, which is ABC News.

Finally, the web-only outlets in the study included The Huffington Post and The Daily Caller, two online-only outlets that have carved distinct identities with a political point of view. The Daily Caller, which is not a part of PEJ’s NCI sample, was not included in the comparisons between legacy media and Twitter for reasons of consistency. (The Daily Caller was included in the study to provide a leading conservative counterpoint to The Huffington Post.)

Outlets Included in Study with Twitter Handles

Newspapers	Broadcast	Radio	Online
The New York Times @nytimes	ABC @abc	NPR @nprnews	The Huffington Post @huffingtonpost
The Washington Post @washingtonpost	CNN @cnn		The Daily Caller @DailyCaller
USA Today @usatoday	MSNBC @msnbc
The Wall Street Journal @wsj	Fox News @foxnews
The Arizona Republic @azcentral
The Blade (Toledo) @toledonews

Researchers captured every tweet during the week of February 14-20, 2011, on the main Twitter account from each outlet, 2,969 tweets in all. The week was chosen because it resembled a typical news week. It is conceivable that in another week, one in which a major breaking news event occurred, the Twitter universe among mainstream media outlets would look different. The goal of this study, however, is to assess the nature of Twitter use during a typical week, not an extraordinary one.

In addition, researchers captured the tweets for this same week from two sets of reporters working for these outlets. These included the reporter from each outlet who had the greatest number of followers on Twitter. To determine the reporters at each organization on Twitter, researchers used a number of strategies. They began by identifying reporters by searching for the outlet on MuckRack.com, a website that lists journalists on Twitter by media outlet and links to their Twitter feeds. In addition, researchers found reporter Twitter accounts by searching for the outlet in Twitter and viewing all of the “people” results. Reporters were added to the list if the Twitter biography identified the person as a reporter for the outlet. Researchers also searched for the outlets in Twello, a website that allows users to search Twitter profiles for key terms such as news organization names. Once again, researchers added any feeds that identified the reporter as working for the outlet in question. Finally, researchers also looked at staff lists (if one was available) on the news organizations’ websites and then searched for the names of staff on Twitter. Here researchers included Twitter feeds for those on staff lists regardless of whether their Twitter profile mentioned the news organization. This combination of searches allowed researchers to build as comprehensive a list as possible of reporters on Twitter from the media organizations of interest in late January/early February 2011. Once researchers had identified all reporters on Twitter, they selected the reporter who had the greatest number of followers as of the end of January for inclusion in the study.

In addition, researchers examined the Twitter feed from a health reporter at each organization in order to examine how reporters on a specialty beat use Twitter (researchers were unable to identify distinct health specialists at the Toledo Blade or The Daily Caller). In the event that an outlet had more than one health reporter, researchers selected for analysis the reporter who had the greater number of followers.

The list of specific reporters included in the study along with their Twitter handles can be found in the table below. There were 677 reporter tweets studied in all.

List of Reporters with Twitter Handles That Were Included in Study

News Organization	Most Followed Reporter	Health Reporter
The Washington Post	Steven Goff (Sports reporter focused on soccer) @soccerinsider	Jennifer LaRue Huget @jhuget
The New York Times	David Pogue (Technology columnist) @Pogue	Tara Parker-Pope @nytimeswell
USA Today	Whitney Matheson (Pop culture blogger) @popcandy	Liz Szabo @LizSzabo
The Wall Street Journal	Kara Swisher (Columnist for All Things Digital) @karaswisher	Katherine Hobson @KatherineHobson
The Arizona Republic	Paul Coro (Sports reporter) @paulcoro	Ken Alltucker @kalltucker
Toledo Blade	Ryan Autullo (Sports reporter) @RyanAutullo
CNN	Jack Gray @jackgraycnn	Dr. Sanjay Gupta @sanjayguptaCNN
MSNBC	Rachel Maddow @maddow	Melissa Dahl @melissadahl
Fox News	Megyn Kelly @megynkelly	Dr. Manny Alvarez @drmannyonFOX
ABC	George Stephanopoulos @gstephanopoulos	Dr. Richard Besser @DrRichardBesser
NPR	Scott Simon @nprscottsimon	Scot Hensley @scotthensley
The Huffington Post	Arianna Huffington @ariannahuff	Dr. Patricia Fitzgerald @DrPatriciaFitz
The Daily Caller	Tucker Carlson @TuckerCarlson

Audit

In order to understand the broader presence on Twitter, beyond the main news feed of the news outlets in the study, researchers conducted an audit in October 2011.

First, researchers tallied the number of Twitter feeds associated with each outlet. Defunct feeds were not included, nor were reporter feeds. (However, television programs that are branded around an individual were included, for instance, @hannityshow, but not @rachelmaddow.)

Because some outlets did not provide a Twitter directory, slightly varying methods were used to tally the number of feeds.

For the following outlets, researchers relied on an official directory posted on their site:

The New York Times
The Wall Street Journal
The Arizona Republic
ABC News
USA Today

For the following outlets, researchers relied on a combination of Twitter lists curated by the organization itself, searches on Twitter, and third-party lists such as those provided by Listorious:

NPR
The Daily Caller
The Blade (Toledo)
CNN
Fox News Channel
MSNBC
The Washington Post

For the Huffington Post, a careful search of each section on the website yielded its universe of Twitter feeds.

The audit yielded a total of 527 feeds.

Next, a snapshot of Twitter activity was taken by analyzing each of those 527 Tweets during the space of a single day. On October 6, 2011, researchers tallied the number of followers listed, and then tallied the number of tweets that were posted on October 4. October 4 was a relatively typical news day.

Researchers had originally recorded the number of followers for each of the outlets’ main Twitter news feed on February 21, 2011 (except for The Arizona Republic, whose number of followers was recorded on February 25). This number was used to calculate the percentage growth in followers between February and October 2011.

Coding

[3]

In order to better understand how Twitter is used by legacy news organizations, tweets were also coded for whether they sought information, were retweets, contained links and, in the case of individual reporters, contained personal anecdotes. Coders noted if the tweet was a retweet and, if so, whether the source of the original tweet was a reporter or another type of Twitter feed at the same news organization, at another news organization, or some other type of source. To understand how much Twitter is used for promotion and driving site traffic, coders noted if the tweet contained a link and characterized the content the link took a reader to. Specifically, researchers recorded whether the link was to news, editorial or commentary, or some other type of content and whether that content was on the news organization’s own website or came from outside the news organization being coded. Every tweet was also coded for whether or not it explicitly sought information from followers. This would include seeking quotes or sources for a story, seeking opinion from followers or asking followers to provide some kind of response or feedback. Finally, in the case of tweets from individual reporters, researchers also coded for whether the tweet contained a personal anecdote. The question here was whether the reporter was relating a personal anecdote about their own life such as discussing what they were doing or commenting on their life.

Every tweet was independently coded by two separate trained undergraduate coders from the School of Media and Public Affairs at The George Washington University. Coders were assigned a set of tweets from one or more news organizations and reporters. Coding was done over one month in late March and April of 2011. If intercoder reliability for a pair of coders on a given variable was 90% or above, the study principals randomly determined which coder’s results to use for the final dataset. If intercoder reliability was less than 90% for a given pair on a given variable, a PEJ or SMPA researcher coded the tweets to reconcile differences between coders and make a final coding decision. Intercoder reliability prior to resolution by the third coder ranged from 77% to 100%.

Variable	Average Intercoder Reliability*
Number people who retweeted (range)	97.0%
Big Story	89.0%
Geographic Focus	79.2%
Personal Anecdote	93.1%
Information Gathering	96.9%
Retweet	98.4%
Retweet Source	97.2%
Tweet Contain Link	99.7%
Driving Site Traffic (characterize link)	77.2%

*Pairwise percent agreement across 15 pairs of coders

In resolving discrepancies, PEJ and SMPA researchers recoded the first 30 tweets where there was disagreement between coders. If, for a given variable, the third coder agreed with one student in the pair 90% or more of the time, that student coder’s data was used. If the third coder did not agree with one student 90% of the time on the first 30 discrepancies, the coder recoded all tweets where the initial pair disagreed to resolve discrepancies.

[1]

[3]