How encouragement language or an informative callout affect the number of comments left Experiment ID: #9640
CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.
Ended On: 11/7/2018
CaringBridge sought to increase engagement on their site by increasing the number of comments on journal posts. Not only did a comment increase the likelihood that someone would return, it actually provided a positive boost to the patient or author as well who was writing the journal. Their standard comment functionality just said “comments”, and didn’t have a true call to action. While someone could scroll to read the comments and there was a box, there was no incentive for them to actually leave a comment at the end of the post. The team developed two treatments: one that provided an “informative fact” that told them what the end result of the comment was to the reader: “Did you know? A quick comment, no matter the situation (positive or negative), shows your support”. This was designed to encourage people to comment regardless of the news. Their second treatment was designed to get people into the comments, using the language “See comments”, and then implicitly encouraged them to “Show [their] love and support” by doing the same. They launched the two treatments in an A/B/C test, monitoring
How will encouragement language or an informative callout affect the number of comments left?
|Treatment Name||Conv. Rate||Relative Difference||Confidence|
|T2:||Comment Dropdown Link||4.1%||-0.0%||3.1%|
This experiment has a required sample size of 25,039 in order to be valid. Since the experiment had a total sample size of 1,949,356, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 10.5% increase in conversion rate
× 0% increase in average gift
The second treatment, with the encouragement language, produced no lift at all. But the “informative fact” treatment produced a valid 10% lift in the number of comments left on journal posts. This suggests that informing the user of the future impact of the intended action can result in more completions of that action. It also shows that removing the implicit objection (in this case, that the journal update was positive or negative) can increase the amount of engagements.
This initial positive test shows great potential to further improve the number of comments left on CaringBridge journal posts.