CaringBridge

How encouragement language or an informative callout affect the number of comments left

Experiment ID: #7578

CaringBridge

CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.

Experiment Summary

Ended On: 11/07/2018

CaringBridge sought to increase engagement on their site by increasing the number of comments on journal posts. Not only did a comment increase the likelihood that someone would return, it actually provided a positive boost to the patient or author as well who was writing the journal. Their standard comment functionality just said “comments”, and didn’t have a true call to action. While someone could scroll to read the comments and there was a box, there was no incentive for them to actually leave a comment at the end of the post. The team developed two treatments: one that provided an “informative fact” that told them what the end result of the comment was to the reader: “Did you know? A quick comment, no matter the situation (positive or negative), shows your support”. This was designed to encourage people to comment regardless of the news. Their second treatment was designed to get people into the comments, using the language “See comments”, and then implicitly encouraged them to “Show [their] love and support” by doing the same. They launched the two treatments in an A/B/C test, monitoring

Research Question

How will encouragement language or an informative callout affect the number of comments left?

Design

C: Original Comment
T1: Quick Fact
T2: Comment Dropdown Link

Results

 Treatment NameClick RateRelative DifferenceConfidence
C: Original Comment 4.1%
T1: Quick Fact 4.5%10.5% 100.0%
T2: Comment Dropdown Link 4.1%0.0% 3.1%

This experiment has a required sample size of 25,039 in order to be valid. Since the experiment had a total sample size of 1,949,356, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    10.5% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The second treatment, with the encouragement language, produced no lift at all. But the “informative fact” treatment produced a valid 10% lift in the number of comments left on journal posts. This suggests that informing the user of the future impact of the intended action can result in more completions of that action. It also shows that removing the implicit objection (in this case, that the journal update was positive or negative) can increase the amount of engagements.

This initial positive test shows great potential to further improve the number of comments left on CaringBridge journal posts.


Experiment Documented by Jeff Giddens
Jeff Giddens is President of NextAfter.

Question about experiment #7578

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.