How anniversary language and monthly donor language affect conversion rate on a monthly giving popup

Experiment ID: #118398


CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.

Experiment Summary

Ended On: 12/08/2022

CaringBridge had used popups to ask repeat visitors to consider giving a monthly gift. While the performance had built a file of recurring donors, they wanted to see if they could increase the clickthrough rate and conversion rate to monthly donor. They formed two hypotheses:

  1. Using language highlighting their 25th anniversary would increase relevance and clickthrough rate and conversion rate
  2. Using “monthly donor” rather than “CaringBridge partner” would increase the clarity of the ask and increase conversion rate.


They decided to test this with an A/B/C test. Their first treatment introduced the 25th anniversary language, but kept the “partner” language. The second kept the 25th anniversary language, but introduced “monthly donor” in place of “partner”.

They launched this to determine a winning variant.

Research Question

We believe that adding anniversary language and changing “partner” to “monthly donor” for returning visitors will achieve an increase in conversion rate.


C: Control
T1: Treatment #1
T2: Treatment #2


  Treatment Name Conv. Rate Relative Difference Confidence
C: Control 0.00%
T1: Treatment #1 0.01% 39.9% 28.1%
T2: Treatment #2 0.01% 184.4% 77.9%

This experiment has a required sample size of 120,900 in order to be valid. Since the experiment had a total sample size of 166,626, and the level of confidence is not above 95% the experiment results are not valid.

Key Learnings

While there weren’t enough monthly transactions to validate the experiment, the second treatment (which included both the added relevance of the 25th anniversary language and the clarity of the term “monthly donor”) showed a non-valid lift of 42%.

The low conversion rate made this unlikely to validate, as the difference was 0.00008205212361% for the control and 0.0001166588894 for the second treatment. However, it was clear that the first treatment underperformed, so they decided to run this experiment with only two variants to see if they could deliver a valid result.

Experiment Documented by Jeff Giddens
Jeff Giddens is President of NextAfter.

Question about experiment #118398

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.