How anniversary language and monthly donor language affect conversion rate on a monthly giving popup
CaringBridge
CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.
Experiment Summary
Ended On: 12/08/2022
CaringBridge had used popups to ask repeat visitors to consider giving a monthly gift. While the performance had built a file of recurring donors, they wanted to see if they could increase the clickthrough rate and conversion rate to monthly donor. They formed two hypotheses:
- Using language highlighting their 25th anniversary would increase relevance and clickthrough rate and conversion rate
- Using “monthly donor” rather than “CaringBridge partner” would increase the clarity of the ask and increase conversion rate.
They decided to test this with an A/B/C test. Their first treatment introduced the 25th anniversary language, but kept the “partner” language. The second kept the 25th anniversary language, but introduced “monthly donor” in place of “partner”.
They launched this to determine a winning variant.
Research Question
We believe that adding anniversary language and changing “partner” to “monthly donor” for returning visitors will achieve an increase in conversion rate.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.00% | ||
| T1: | Treatment #1 | 0.01% | 39.9% | 28.1% |
| T2: | Treatment #2 | 0.01% | 184.4% | 77.9% |
This experiment has a required sample size of 120,900 in order to be valid. Since the experiment had a total sample size of 166,626, and the level of confidence is not above 95% the experiment results are not valid.
Key Learnings
While there weren’t enough monthly transactions to validate the experiment, the second treatment (which included both the added relevance of the 25th anniversary language and the clarity of the term “monthly donor”) showed a non-valid lift of 42%.
The low conversion rate made this unlikely to validate, as the difference was 0.00008205212361% for the control and 0.0001166588894 for the second treatment. However, it was clear that the first treatment underperformed, so they decided to run this experiment with only two variants to see if they could deliver a valid result.