Methodology

The study, The Media and Herman Cain, uses content analysis data from two sources. Data regarding the quantity of coverage is mostly derived from the Project for Excellence in Journalism’s in-house coding operation. (Click here for details on how that project, also known as PEJ’s News Coverage Index, is conducted.)

To arrive at the results regarding the tone of coverage, PEJ employed a combination of traditional media research methods, based on long-standing rules regarding content analysis, along with computer coding software developed by Crimson Hexagon. That software is able to analyze the textual content from billions of messages on blogs, Twitter, Facebook and web-based articles from news sites. Crimson Hexagon (CH) classifies online content by identifying statistical patterns in words.

Use of Crimson Hexagon’s Technology

The technology is rooted in an algorithm created by Gary King, a professor at Harvard University’s Institute for Quantitative Social Science. (Click here to view the study explaining the algorithm.)

The purpose of computer coding in general, and Crimson Hexagon specifically, is to “take as data a potentially large set of text documents, of which a small subset is hand coded into an investigator-chosen set of mutually exclusive and exhaustive categories. As output, the methods give approximately unbiased and statistically consistent estimates of the proportion of all documents in each category.”

Universe

Crimson Hexagon software examines online content provided by RSS feeds of thousands of news outlets from the U.S. and around the world. This provides researchers with analysis of a much wider pool of content than conventional human coding can provide. Specifically, the monitors PEJ created for this study are based on more than 11,500 news web sites. CH maintains a database of all stories available so texts can be investigated retroactively.

While the software collects and analyzes online content, the database includes many news sites produced by television and radio outlets. Most stations do not offer exact transcripts of their broadcasted content on their sites and RSS feeds, however, those sites often include text stories that are very similar to report that were aired. For example, even though the television programs from Fox News are not in the sample directly, content from Fox News is present through the stories published on FoxNews.com.

The universe includes content from all the major television networks along with thousands of local television and radio stations. Two notable television sources, CBS and PBS’ NewsHour, do offer transcripts of their television news programs, and those texts are including in the sample.

Elite vs. Broad Sample

For this report, PEJ examined two different samples using Crimson Hexagon’s database of news outlets. The universe described as the “broad” sample includes all of the more than 11,500 news sites available. Not all of these outlets contain campaign stories on a regular basis, but any time they do, those stories are included in the sample. For instance, local television newscasts may not offer much coverage of the presidential campaign. However, the sample will include any relevant reports that do appear.

The universe entitled the “elite” sample is made up of a smaller collection of news sites that provide a focused cross section of national media based in large part on audience numbers. This elite sample is based on the 52 different outlets included in PEJ’s weekly News Coverage Index (NCI) which includes print, cable, radio, online and broadcast.

Of the 52 outlets found in the NCI, 47 are included in the elite sample. For technical reasons, five sources cannot be represented. Three radio talk shows (Rush Limbaugh, Sean Hannity and Ed Schultz) do not have accompanying RSS feeds. The Wall Street Journal is not included in the algorithmic tone coding due to its paywall. Google News, which does not produce original material but rather pulls stories from other sources, is not included because the same material is coded in the other outlets where it appears.

The 47 outlets’ content is distributed through 21 unique URLs. This number is smaller because television websites often serve as umbrellas for web feeds from multiple programs. For example, the URL abcnews.go.com includes feeds from both Good Morning America and ABC’s World News with Diane Sawyer. Foxnews.com provides material from Fox News programs including Special Report with Bret Baier, Fox Report with Shepard Smith, the O’Reilly Factor and Hannity.

The list of URLs that are included in the elite sample are as follows:

• 1. http://www.msnbc.msn.com/
• 2. http://www.today.msnbc.msn.com/
• 3. http://www.ed.msnbc.msn.com/
• 4. http://www.cnn.com/
• 5. http://www.foxnews.com/
• 6. http://www.abcnews.go.com/
• 7. http://www.cbsnews.com/
• 8. http://www.npr.org/
• 9. www.pbs.org/newshour
• 10. http://www.nytimes.com/
• 11. http://www.washingtonpost.com/
• 12. http://www.usatoday.com/
• 13. http://www.ajc.com/
• 14. http://www.latimes.com/
• 15. http://www.toledoblade.com/
• 16. http://www.azcentral.com/
• 17. http://www.thehour.com/
• 18. http://www.spokesman.com/
• 19. http://www.joplinglobe.com/
• 20. http://www.news.yahoo.com/
• 21. http://www.huffingtonpost.com/

Monitor Creation and Training

Each individual study or query related to a set of variables is referred to as a “monitor.”

The process of creating a new monitor consists of four steps. (See below for an example of these steps in action.)

First, PEJ researchers decide what timeframe and universe of content to examine – general news stories, blogs, messages on the major social media sites Twitter and Facebook or some combination. For this study, the focus was solely on English-language news outlets.

Second, the researchers enter key terms using Boolean search logic so the software can identify the universe of posts to analyze.

Next, researchers define categories appropriate to the parameters of the study. If a monitor is measuring the tone of coverage for a specific politician, for example, there would be four categories: positive, neutral, negative, and irrelevant for posts that are off-topic in some way.

If a monitor is measuring media framing or storyline, the categories would be more extensive. For example, a monitor studying the framing of coverage about the death of Osama bin Laden might include nine categories: details of the raid, global reaction, political impact, impact on terrorism, role of Pakistan, straight account of events, impact on U.S. policy, the life of bin Laden, and a category off-topic posts.

Fourth, researchers “train” the CH platform to analyze content according to specific parameters they want to study. The PEJ researchers in this role have gone through in-depth training at two different levels. They are professional content analysts fully versed in PEJ’s existing content analysis operation and methodology. They then undergo specific training on the CH platform including multiple rounds of reliability testing.

The monitor training itself is done with a random selection of posts collected by the technology. One at a time, the software displays posts and a human coder determines which category each example best fits into. In categorizing the content, PEJ staff follows coding rules created over the many years that PEJ has been content analyzing news media. If an example does not fit easily into a category, that specific post is skipped. The goal of this training is to feed the software with clear examples for every category.

For each new monitor, human coders categorize at least 250 distinct posts. Typically, each individual category includes 20 or more posts before the training is complete. To validate the training, PEJ has conducted numerous intercoder reliability tests (see below) and the training of every monitor is examined by a second coder in order to discover errors.

The training process consists of researchers showing the algorithm stories in their entirety that are unambiguous in tone. Once the training is complete, the algorithm analyzes content at the assertion level, to ensure that the meaning is similarly unambiguous. This makes it possible to analyze and proportion content that contains assertions of differing tone. This classification is done by applying statistical word patterns derived from posts categorized by human coders during the training process

How the Algorithm Works

To understand how the software recognizes and uses patterns of words to interpret texts, consider a simplified example. Imagine the study examining coverage regarding the death of Osama bin Laden that utilizes the nine categories listed above. As a result of the example stories categorized by a human coder during the training, the CH monitor might recognize that portions of a story with the words “Obama,” “poll” and “increase” near each other are likely about the political ramifications. However, a section that includes the words “Obama,” “compound” and “Navy” is likely to be about the details of the raid itself.

Unlike most human coding, CH monitors do not measure each story as a unit, but examine the entire discussion in the aggregate. To do that, the algorithm breaks up all relevant texts into subsections. Rather than the dividing each story, paragraph, sentence or word, CH treats the “assertion” as the unit of measurement. Thus, posts are divided up by the computer algorithm. If 40% of a story fits into one category, and 60% fits into another, the software will divide the text accordingly. Consequently, the results are not expressed in percent of newshole or percent of stories. Instead, the results are the percent of assertions out of the entire body of stories identified by the original Boolean search terms. We refer to the entire collection of assertions as the “conversation.”

Testing and Validity

Extensive testing by Crimson Hexagon has demonstrated that the tool is 97% reliable, that is, in 97% of cases analyzed, the technology’s coding has been shown to match human coding. PEJ spent more than 12 months testing CH and its own tests comparing coding by humans and the software came up with similar results.

In addition to validity tests of the platform itself, PEJ conducted separate examinations of human intercoder reliability to show that the training process for complex concepts is replicable. The first test had five researchers each code the same 30 stories which resulted in an agreement of 85%.

A second test had each of the five researchers build their own separate monitors to see how the results compared. This test involved not only testing coder agreement, but also how the algorithm handles various examinations of the same content when different human trainers are working on the same subject. The five separate monitors came up with results that were within 85% of each other.

Unlike polling data, the results from the CH tool do not have a sampling margin of error since there is no sampling involved. For the algorithmic tool, reliability tested at 97% meets the highest standards of academic rigor.

Ongoing Monitors

In some instances, PEJ uses CH to study a given period of time, and then expand the monitor for additional time going forward. In order to accomplish this, researchers first create a monitor for the original timeframe according to the method described above.

Because the tenor and content of online conversation can change over time, additional training is necessary if the timeframe gets extended. Since the specific conversation about candidates evolves all the time, the CH monitor must be trained to understand how newer posts fit into the larger categories.

In those instances, researchers conduct additional training for the monitor with a focus on posts that occurred during the new time period. For every new week that is examined, at least 25 more posts are added to the monitor’s training. At that point, the monitor is run to come up with new results for the expanded time period which are added to results that were already derived in the original timeframe.

An Example

Since the use of computer-aided coding is a relatively new phenomenon, it will be helpful to demonstrate how the above procedure works by following a specific example.

PEJ created a monitor to measure the tone of media coverage on news sites for Republican candidate Mitt Romney. First, we created a monitor with the following guidelines:

• 1. Source: “News” sources only
• 2. Original date range: May 2 to September 11, 2011
• 3. English-language content only
• 4. Keyword: Romney

We then created the four categories that are used for measuring tone:

• 1. Positive
• 2. Neutral
• 3. Negative
• 4. Off-topic/Irrelevant

Next, we trained the monitor by classifying documents. CH randomly selected entire posts from the time period specified, and displayed them one by one. A PEJ researcher decided if each post is a clear example of one of the four categories, and if so, assigned that post into the appropriate category. If an example post is not clear in its meaning, or could fit into more than one category, such as a story with a mix of positive and negative assertions, the coder skipped the post. Since the goal is to find the clearest cases possible, coders will often skip many posts until they find good examples.

A story that is entirely about a poll showing Mitt Romney ahead of the Republican field – and that his lead is growing, would be a good example to put in the “positive” category. A different story that is entirely about Romney’s record in Massachusetts and how many conservative voters are opposed to him would be put in the “negative” category. A post that is strictly factual, such as a story about a speech Romney gave on the economy that does not include evaluative assessments, would be put in the “neutral” category. And a post that includes the word “Romney” but is not about the candidate at all, such as a story about a different person with the same last name, would go in the “off-topic” category.

The coder trained 260 documents in all – ten more than the necessary minimum of 250. Each of the four categories had more than 20 posts in them.

At that point, the initial training was finished. For the sake of validity, PEJ has another coder check over all of our training and look for stories that they would have categorized differently. Those stories are removed from the training sample because the disagreement between coders shows that they are not clear, precise examples. In the case of the Romney monitor, there were four documents that were removed for this reason.

Finally, we “ran” the monitor. This means that the algorithm examined the word patterns derived from the monitor training, and applied those patterns every post that was captured using the initial guidelines. Since the software studies the conversation in an aggregate as opposed to individual posts or stories, the algorithm divided up the overall conversation into percentages that fit into the four categories.

For the initial monitor, the algorithm examined over 94,000 assertions from thousands of news stories and determined that 34% of the conversation was positive, 33% neutral, and 33% negative. The assertions or statements that are off-topic were excluded from the results.

In order to extend the Romney monitor beyond September 11, coders added at least 25 new pieces of content to the training for each new week examined. This assures that any linguistic changes in the overall coverage or conversation regarding Romney in the new week are accounted for. We then run the monitor again, which now includes the original training of 260 posts plus 25 new ones, for the new week while leaving the earlier results in place.