How a message box on the front end of a donation ask affects conversion
CaringBridge
CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.
Experiment Summary
Timeframe: 06/03/2020 - 06/11/2020
CaringBridge’s tribute widget was one of the primary channels of donations on their site. Appearing at the bottom of every person’s homepage and journal, the tribute widget made a donation ask, and then gave the user the option to enter a message of support to go along with their gift. The CaringBridge team had always wanted to know whether this was a “feature or a bug”. Did the message box improve conversion by getting more people to start their gift by writing a message—or did the large box intimidate potential donors and turn them away? They decided to simply remove the box and see what happened. Without it, there was just a standalone donation ask with a button. They ran the treatment to just 20% of the audience to mitigate potential risk, since this got enough traffic to validate with a small window of time.
Research Question
How will removing a message box on the front end of a donation ask affect conversion?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | 0.69% | $0.00 | ||
| T1: | No message box | 0.50% | -28.3% | 99.1% | $0.00 |
This experiment has a required sample size of 12,978 in order to be valid. Since the experiment had a total sample size of 58,366, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 28.3% decrease in conversion rate
× 0% increase in average gift
Key Learnings
Initially, the treatment showed a small lift. But given more time, the treatment began to show decreased results. The overall results showed around a 10% decrease in conversion, but it was slow to validate. Drilling down into segments revealed why: desktop was a 28.3% decrease, totally valid. But mobile devices showed little effect, though it constituted the majority of the traffic. That held the overall experiment from validating, and obscured the real finding: removing the message box lowers desktop donations.
The hypothesis behind this is that desktop visitors are more likely at a keyboard and able to type a message, which is where the message box truly has impact. If there was a lift on mobile, then it might suggest the opportunity for two separate experiences, but indicators showed average gift was slightly down on mobile. This adds a second layer of hypothesis: writing a message spurs an emotional response that causes people to give more in honor of their friend.
Both of these indicators suggest that the message box should stay, and testing should extend to other areas.