How communal language affects clickthrough rate on a recurring giving popup
CaringBridge
CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.
Experiment Summary
Ended On: 02/09/2021
CaringBridge had run a popup asking visitors on their 2nd and 8th visits to become a recurring donor for more than a year, which had built a strong base of monthly donors. The control asked donors to “consider something” before they go, and then talked about the need for recurring donors. They decided to try to include some “communal language”, that first thanked the donor for staying connected to the CaringBridge community and then solicited them for a monthly gift based on their firsthand experience of the value of the CaringBridge community.
They decided to test this new language through an A/B test.
Research Question
We believe that employing communal language for repeat visitors will achieve an increase in monthly donors.
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.73% | ||
| T1: | Treatment #1 | 0.86% | 18.7% | 98.6% |
This experiment has a required sample size of 31,992 in order to be valid. Since the experiment had a total sample size of 120,105, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
18.7% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The communal language on the popup increased clickthrough rate by 18.7%. However, upon reviewing the conversion rate on the donation page from the two popup treatments, it was clear that the communal language attracted more clicks but decreased the motivation of the donors, as conversion rate dropped 75.2% from the increase in clicks. This is a reminder that all clicks are not created equal—more traffic does not necessarily result in more donors if those donors are not properly motivated and incentivized by the existing copy.
CaringBridge had a secondary hypothesis that the language on the donation page was not robust enough, and, with some improvement to the page copy, might be able to convert more of these “communal language clickers”. They decided to run a monthly giving donation page test first and then re-test this with the winner.