Methodology

The study, “Twitter and the Campaign” uses content analysis data from two sources.

Data regarding the quantity of coverage in the traditional press is derived from the Project for Excellence in Journalism’s in-house coding operation. (Click here for details on how that project, also known as PEJ’s News Coverage Index, is conducted.)

To arrive at the results regarding the quantity of coverage on blogs and Twitter, and the tone of coverage on all outlets, PEJ employed a combination of traditional media research methods, based on our long-standing rules regarding content analysis, along with computer coding software developed by Crimson Hexagon. That software is able to analyze the textual content from billions of messages on blogs, Twitter and web-based articles from news sites. Crimson Hexagon (CH) classifies online content by identifying statistical patterns in words.

Use of Crimson Hexagon’s Technology

The technology is rooted in an algorithm created by Gary King, a professor at Harvard University’s Institute for Quantitative Social Science. (Click here to view the study explaining the algorithm.)

The purpose of computer coding in general, and Crimson Hexagon specifically, is to “take as data a potentially large set of text documents, of which a small subset is hand coded into an investigator-chosen set of mutually exclusive and exhaustive categories. As output, the methods give approximately unbiased and statistically consistent estimates of the proportion of all documents in each category.”

Universe

Crimson Hexagon software examines online content provided by RSS feeds of millions of sites from the U.S. and around the world. This provides researchers with analysis of a much wider pool of content than conventional human coding can provide. CH maintains a database of all texts available so content can be investigated retroactively.

Twitter

Crimson Hexagon draws its universe of tweets from something called the “Twitter Firehose data feed.” That is a feed of all the tweets on the twitter platform that are public. According to the Twitter’s own blog, there are about 140 million tweets posted each day that are public on the Firehose feed. (The Firehose does not include private tweets. However, since private tweets are sent to individuals, much like emails, they are not part of public conversations.)

The volume of conversation in Twitter is referred to as “assertions.” The number of assertions refers to the quantity of statements or opinions focused on each person.

Because CH examines text in the aggregate, it is not enough to simply count the number of tweets where a person’s name shows up to gauge how often a candidate is being discussed. Some tweets may include multiple statements or opinions, while others may use the same word as a candidate’s name without referring to the candidate. For example, a tweet that refers to the Huntsman Center at the University of Utah is likely unrelated to the presidential campaign of Jon Huntsman, and is therefore discarded from the sample studied in this report.

Therefore, the number of assertions is a more accurate measure because it includes the relevant statements about the subjects in the race without extraneous information.

Blogs

Crimson Hexagon’s sample includes hundreds of thousands of blogs from the U.S. and around the world. While the blogosphere is growing every day, it is impossible to create a sample that includes all possible blogs. However, CH’s universe is one of the largest blog samples available and includes all of the most popular English-language political and news blogs.

As with Twitter, the volume of conversation is referred to as “assertions.” The number of assertions refers to the quantity of statements or opinions focused on each person. Since blog posts can range in size, many posts include multiple assertions. At the same time, other long blog posts focused on a non-campaign subject may mention a candidate in a single sentence, and the software learns to include only that relevant statement while exempting the rest from the sample.

Traditional Press

The monitors focused on the traditional press are based on more than 11,500 news websites.

While the software collects and analyzes online content, the database includes many news sites produced by television and radio outlets. Most stations do not offer exact transcripts of their broadcast content on their sites and RSS feeds, however, those sites often include text stories that are very similar to reports that were aired. For example, even though the television programs from Fox News are not in the sample directly, content from Fox News is present through the stories published on FoxNews.com.

The universe includes content from the websites of all the major television networks, such as ABC, NBC, MSNBC and CNN, along with thousands of local television and radio stations. Two notable television sources, CBS and PBS’ NewsHour, do offer transcripts of their television news programs, and those texts are included in the sample.

Elite vs. Broad Sample in the Traditional Press

For this report, PEJ examined two different samples using Crimson Hexagon’s database of news outlets. Most of the discussion is focused on the universe described as the “broad” sample which includes all of the more than 11,500 news sites available. Not all of these outlets contain campaign stories on a regular basis, but any time they do, those stories are included in the sample. For instance, local television newscasts may not offer much coverage of the presidential campaign. However, the sample will include any relevant reports that do appear.

The universe entitled the “elite” sample is made up of a smaller collection of news sites that provide a focused cross section of national media based in large part on audience numbers. This elite sample is based on the 52 different outlets included in PEJ’s weekly News Coverage Index (NCI) which includes print, cable, radio, online and broadcast.

Of the 52 outlets found in the NCI, 47 are included in the elite sample. For technical reasons, five sources cannot be represented. Three radio talk shows (Rush Limbaugh, Sean Hannity and Ed Schultz) do not have accompanying RSS feeds. The Wall Street Journal is not included in the algorithmic tone coding due to its paywall. Google News, which does not produce original material but rather pulls stories from other sources, is not included because the same material is coded in the other outlets where it appears.

The 47 outlets’ content is distributed through 21 unique URLs. This number is smaller because television websites often serve as umbrellas for web feeds from multiple programs. For example, the URL abcnews.go.com includes feeds from both Good Morning America and ABC’s World News with Diane Sawyer. Foxnews.com provides material from Fox News programs including Special Report with Bret Baier, Fox Report with Shepard Smith, the O’Reilly Factor and Hannity.

The list of URLs that are included in the elite sample are as follows:

Monitor Creation and Training

Each individual study or query related to a set of variables is referred to as a “monitor.”

The process of creating a new monitor consists of four steps. (See below for an example of these steps in action.)

First, PEJ researchers decide what timeframe and universe of content to examine-general news stories, blogs, messages on the major social media sites Twitter and Facebook or some combination. For this study, the focus was solely on English-language tweets, blogs and news outlets.

Second, the researchers enter key terms using Boolean search logic so the software can identify the universe of posts to analyze.

Next, researchers define categories appropriate to the parameters of the study. If a monitor is measuring the tone of tweets or coverage for a specific politician, for example, there would be four categories: positive, neutral, negative, and irrelevant for posts that are off-topic in some way.

If a monitor is measuring media framing or storyline, the categories would be more extensive. For example, a monitor studying the framing of coverage about the death of Osama bin Laden might include nine categories: details of the raid, global reaction, political impact, impact on terrorism, role of Pakistan, straight account of events, impact on U.S. policy, the life of bin Laden, and a category off-topic posts.

Fourth, researchers “train” the CH platform to analyze content according to specific parameters they want to study. The PEJ researchers in this role have gone through in-depth training at two different levels. They are professional content analysts fully versed in PEJ’s existing content analysis operation and methodology. They then undergo specific training on the CH platform including multiple rounds of reliability testing.

The monitor training itself is done with a random selection of posts collected by the technology. One at a time, the software displays posts and a human coder determines which category each example best fits into. In categorizing the content, PEJ staff follows coding rules created over the many years that PEJ has been content analyzing news media. If an example does not fit easily into a category, that specific post is skipped. The goal of this training is to feed the software with clear examples for every category.

For each new monitor, human coders categorize at least 250 distinct posts. Typically, each individual category includes 20 or more posts before the training is complete. To validate the training, PEJ has conducted numerous intercoder reliability tests (see below) and the training of every monitor is examined by a second coder in order to discover errors.

The training process consists of researchers showing the algorithm stories in their entirety that are unambiguous in tone. Once the training is complete, the algorithm analyzes content at the assertion level, to ensure that the meaning is similarly unambiguous. This makes it possible to analyze and proportion content that contains assertions of differing tone. This classification is done by applying statistical word patterns derived from posts categorized by human coders during the training process.

How the Algorithm Works

To understand how the software recognizes and uses patterns of words to interpret texts, consider a simplified example. Imagine the study examining news and blog coverage regarding the death of Osama bin Laden that utilizes the nine categories listed above. As a result of the example stories categorized by a human coder during the training, the CH monitor might recognize that portions of a story with the words “Obama,” “poll” and “increase” near each other are likely about the political ramifications. However, a section that includes the words “Obama,” “compound” and “Navy” is likely to be about the details of the raid itself.

Unlike most human coding, CH monitors do not measure each story, blog or tweet as a unit, but examine the entire discussion in the aggregate. To do that, the algorithm breaks up all relevant texts into subsections. Rather than the dividing each story, paragraph, sentence or word, CH treats the “assertion” as the unit of measurement. (Because of the 140 character limit, most tweets consist of a single assertion but this is not a requirement; if a tweet contains two separate assertions, the monitor will assess each one individually.) Thus, posts are divided up by the computer algorithm. If 40% of a story fits into one category, and 60% fits into another, the software will divide the text accordingly. Consequently, the results are not expressed in percent of newshole or percent of stories. Instead, the results are the percent of assertions out of the entire body of stories identified by the original Boolean search terms. We refer to the entire collection of assertions as the “conversation.”

Testing and Validity

Extensive testing by Crimson Hexagon has demonstrated that the tool is 97% reliable, that is, in 97% of cases analyzed, the technology’s coding has been shown to match human coding. PEJ spent more than 12 months testing CH and its own tests comparing coding by humans and the software came up with similar results.

In addition to validity tests of the platform itself, PEJ conducted separate examinations of human intercoder reliability to show that the training process for complex concepts is replicable. The first test had five researchers each code the same 30 stories which resulted in an agreement of 85%.

A second test had each of the five researchers build their own separate monitors to see how the results compared. This test involved not only testing coder agreement, but also how the algorithm handles various examinations of the same content when different human trainers are working on the same subject. The five separate monitors came up with results that were within 85% of each other.

Unlike polling data, the results from the CH tool do not have a sampling margin of error since there is no sampling involved. For the algorithmic tool, reliability tested at 97% meets the highest standards of academic rigor.

Ongoing Monitors

In some instances, PEJ uses CH to study a given period of time, and then expand the monitor for additional time going forward. In order to accomplish this, researchers first create a monitor for the original timeframe according to the method described above.

Because the tenor and content of online conversation can change over time, additional training is necessary if the timeframe gets extended. Since the specific conversation about candidates evolves all the time, the CH monitor must be trained to understand how newer posts fit into the larger categories.

In those instances, researchers conduct additional training for the monitor with a focus on posts that occurred during the new time period. For every new week that is examined, at least 25 more posts are added to the monitor’s training. At that point, the monitor is run to come up with new results for the expanded time period which are added to results that were already derived in the original timeframe.

An Example

Since the use of computer-aided coding is a relatively new phenomenon, it will be helpful to demonstrate how the above procedure works by following a specific example measuring content on Twitter.

PEJ created a monitor to measure the tone of the conversation on Twitter for Republican candidate Newt Gingrich. First, we created a monitor with the following guidelines:

1. Source: All public Twitter messages
2. Original date range: May 2 to October 16, 2011
3. English-language content only
4. Keyword: Gingrich

We then created the four categories that are used for measuring tone:

1. Positive
2. Neutral
3. Negative
4. Off-topic/Irrelevant

Next, we trained the monitor by classifying documents. CH randomly selected entire tweets from the time period specified, and displayed them one by one. A PEJ researcher decided if each post is a clear example of one of the four categories, and if so, assigned that tweet into the appropriate category. If an example post is not clear in its meaning, or could fit into more than one category, such as a tweet with a mix of positive and negative assertions, the coder skipped the post. Since the goal is to find the clearest cases possible, coders will often skip many posts until they find good examples.

A tweet such as the following: “The ideal ticket…Gingrich to educate the American people about true conservatism,” would be a good example to put in the “positive” category for Gingrich. A different tweet that shows disapproval of Gingrich, such as: “Gingrich is the Newt of all evil,” would be put in the “negative” category. A post that is strictly factual, such as a tweet previewing a debate lineup that reads: “Speaker sked: Ron Paul, Michele Bachmann, Newt Gingrich, Rick Santorum, Rick Perry. Starts at 730. 10 mins each,” would be put in the “neutral” category. And a tweet that includes the word “Gingrich” but is not about the candidate at all, such as a story about a different person with the same last name, would go in the “off-topic” category.

The coder trained 250 documents in all. Each of the four categories had more than 20 posts in them.

At that point, the initial training was finished. For the sake of validity, PEJ has another coder check over all of our training and look for tweets that they would have categorized differently. Those tweets are removed from the training sample because the disagreement between coders shows that they are not clear, precise examples. In the case of the Gingrich monitor, there were six documents removed for this reason.

Finally, we “ran” the monitor. This means that the algorithm examined the word patterns derived from the monitor training, and applied those patterns to every tweet that was captured using the initial guidelines. Since the software studies the conversation in an aggregate as opposed to individual tweets, the algorithm divided up the overall conversation into percentages that fit into the four categories.

For the initial monitor, the algorithm examined over 320,000 assertions from thousands of tweets and determined that 15% of the conversation was positive, 35% neutral, and 50% negative. The assertions or statements that are off-topic were excluded from the results.

In order to extend the Gingrich monitor beyond October 16, coders added at least 25 new tweets of content to the training for each new week examined. This assures that any linguistic changes in the overall coverage or conversation regarding Gingrich in the new week are accounted for. We then run the monitor again each week, which now includes the original training of 250 posts plus 25 new ones, for the new week while leaving the earlier results in place.

Topics

Regions & Countries

Formats

Twitter and the Campaign

Methodology

Most Popular

Table of Contents

Table of Contents

Research Topics

Follow Us

Twitter and the Campaign

Methodology

Table of Contents

Table of Contents

Sign up for The Briefing

Sign up for our weekly newsletter

Related

Most Popular

Table of Contents

Table of Contents

Research Topics

Follow Us