How a donor-based appeal (instead of a goal-based appeal) affects conversion rate
CaringBridge
CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.
Experiment Summary
Ended On: 01/27/2019
CaringBridge had a matching gift offer as part of their year-end campaign. They had recently developed a dynamic thermometer that would update as a goal was reached and had leveraged it to increase conversion. However, they wondered if the large goal amount was too vague, and possibly caused too much cognitive friction in the mind of the reader. Did it make the reader wonder if the large goal could be achieved? Did it make the reader do mental math to approximate how much had been raised? Most importantly—did it communicate to the reader that their gift could have an impact?
They decided to run an experiment with a donor-based appeal, breaking their day down into hourly goals and keeping a dynamic countdown of how many donors were needed to meet the hourly goal. Their hypothesis was that smaller goals would bring it more within reach of the donor, and that communicating a goal in terms of how many donations—rather than how much money—was needed would tell the donor how they could materially affect the goal. Notably, the treatment did not include the match language at all.
They launched an A/B test to understand the impact of this treatment.
Research Question
How will a donor-based appeal (instead of a goal-based appeal) affects conversion rate?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Thermometer | 0.39% | ||
| T1: | Power Hour | 0.49% | 25.1% | 98.0% |
This experiment has a required sample size of 34,742 in order to be valid. Since the experiment had a total sample size of 97,403, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
25.1% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The treatment version produced a 25% increase in conversion—at scale, a tremendous increase in revenue. This suggested that the donor-based goal was not only easier to understand, but more appealing. This approach will be tested again in future appeals to refine our understanding of this increase.