How isolating a quantified ask affects clickthrough rate
CaringBridge
CaringBridge offers free personal, protected websites for people to easily share updates and receive support and encouragement from their community during a health journey. Every 7 minutes, a CaringBridge website is created for someone experiencing a health event.
Experiment Summary
Timeframe: 03/15/2017 - 03/27/2017
CaringBridge tested a radical redesign that produced a big lift in donor conversion. This was a valuable finding, but they wanted further clarity on the lift, since two factors were changed. In the first experiment, they changed the headline of their tribute widget and added a “quantified ask”, with the offer that a $30 gift would keep their friend or loved one’s site online for a month.
They were curious which element had the most impact between these two. So they created two treatments: one that reverted to the old headline and included the quantified ask, and one that reverted to the old body copy (without the quantified ask), but kept the winning headline.
They launched an A/B/C test to validate a winner and determine the impact of each element on conversion.
Research Question
When isolated, what impact does the quantified ask have on clickthrough rate?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control (Both elements changed) | 0.27% | ||
| T1: | Quantified Ask only | 0.34% | 26.0% | 98.3% |
| T2: | Headline change only | 0.24% | -10.6% | 71.1% |
This experiment has a required sample size of 33,523 in order to be valid. Since the experiment had a total sample size of 211,116, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 26.0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The first treatment, with just the quantified ask, produced a 26% increase in clickthrough rate. However, in further testing, it failed to produce any increase in conversion and produced a spike in refunds.
The second treatment, with the headline change only, produced a 10% decrease in clickthrough rate, which showed that the cumulative lift in previous testing had more correlation with the presence of the quantified ask, but was boosted by the headline as well.
This prompted further testing to see if they could clarify the message on the donation page to reduce confusion and refund rate.