Sequencing the DNA of the Perfect Comment [Study]

Since the Internet began, user-generated content (UGC) has played an important role in engaging users and encouraging the spread of content. A recent study examined the contribution of comments to the readers' experience, and by deconstructing them down to base attributes has determined the importance of these elements. Thus determining "The Perfect Comment".

The DNA of a Comment

The study, performed by AOL's Consumer Analytics and Research team (Disclosure: the author works for AOL) with Joe Blechman as lead analyst, and Probit Research, began by breaking down the constituent parts of comments into 15 measurable attributes:

IdentityGeographyIcon PreferenceBadgesNumber of FansNumber of favorites / likes / thumbs upSuper UserGrammar levelHumorClarity of thoughtOriginal article criticismFact-based commentLinks to articleAdherence to party lineIrrelevant / off-trackComment Research

Now that the attributes had been defined, the next step was to determine the best way to measure individual comments. The study used a ratings based conjoint with a sample size of 1,351 readers (aged 18+ from a diverse range of age, geography and gender backgrounds) of eight major online news sites:

The Huffington Post (an AOL company)Fox NewsUSA TodayCNNThe New York TimesThe Washington PostMSNBCYahoo

They were asked to select one of four sample articles, whichever one they were most interested in reading, and given 16 comments from the selected article, one at a time, and asked to rate them on a 5-point scale against nine separate criteria, such as the quality of the article, the emotional response the comment generated for them, and whether the comment made them want to read more or even join the conversation.

The Results

The study showed what users care about when they look at comments on your site.

Style

Some of the least important elements were humor and grammar, although no humor and a collegiate level of grammar did move the dial slightly. If your comments are generally below elementary school level, then you may as well turn your commenting features off as this was a fairly large negative.

Substance

Comments performed better if the content was factually based rather than containing abstract notions. Links within the comments also pull the score down slightly. People also much preferred comments that were written with clarity, with obfuscatory or rambling comments dragging the scores downwards quite sharply.

Community Involvement

Users like to see that a commenter is heavily involved within a community, it makes them feel as though the conversation is one that's worthwhile. However, they don't like to see that a user is too involved. In the study, the users were most likely to respond positively to a comment if the comment has 6+ likes / favorites / votes / thumbs (although if this feature doesn't exist it's neutral, as opposed to if it does exist and the comment has 0 likes as that's a negative almost equal to the positive. Users like to see badges on a site for commenters, and they are more likely to interact with a user who has multiple badges, as well as more than 1,000 fans.

The interesting finding here though, is that not having a Super User feature for commenters is seen as a negative, as is the commenter being a super user. However, if it's available and the commenter isn't a super user, then that's, mildly bizarrely, a positive.

Personal Involvement

Users like to see commenters who use their real names; nicknames are a neutral sell; anonymous commenters do deliver negativity. As for avatars, it doesn't really matter much whether you have commenters who use their real picture or some other pictures, just as long as it's not a default icon or blank. Having some form of geo-location data is also a plus, while it doesn't have to be a specific location, at least a general location will help.

Relationship to Content

Given that this study looked at news sites, with articles that had some political leaning to them, there was an element of partisanship. This was shown in negative scoring for anyone who came in with a particular political bent in their comment, but a very high plus for any comments that were deemed to be independent.

What about the tone of the comments as related to the content of the article? Users were more willing to interact with commenters that were critical of the original post, and on a slightly lesser level, those commenters that were neutral. Commenters that praised the original article were a huge turnoff. The last element was that of relevance. As you may expect, relevant comments were a plus, irrelevant comments were a turn-off.

What Does it All Mean?

With the data (see the chart at the end of this section) you can create a spreadsheet and grade the comments on your site. You can compare yourself to your competitors and see who does the better job. You can figure out what needs fine tuning on your site – what features you need to implement, how you can nudge your commenters to interact, and when you should just sit back and let things take their own direction.

For example, if you heavily moderate critical comments, perhaps you should back off doing so, as it may generate more engagement. Perhaps during the sign up process you should encourage users to use their name, to use a picture and to specify some form of location. Adding badges / super user features, and using these to actively encourage friending / liking comments can also be beneficial for your site.

A Disclaimer and a Winner

As this study was performed on the 8 news sites listed above, your community may vary in what they deem to be important. However, given that this was the first study of its kind, it's not a bad starting point for any site looking to improve their community engagement.

As for the 8 sites, when the formula was then applied to 10 comments sampled from 40 article on each of the sites, there was a clear winner. USA Today outscored the competition in each category examined, with Fox News having the most capacity for improvement.

You can download the full study (PDF) here.

Bringing Together Paid, Owned and Earned Media
Sept. 10-13, 2013: With a newly announced, completely renovated agenda,
SES San Francisco could be the most valuable online marketing conference you attend this year. Register today and save up to $200!
*Pre-show rate through September 6.