How will the presence of a value proposition banner in checkout affect conversion?
Compassion International
Experiment Summary
Ended On: 05/21/2019
Compassion International has a three-step checkout process to sponsor a child. They have long known that there is a drop-off rate for each step of the process, and have wanted to optimize that—knowing that people who have already selected a child have a high motivation and should be more likely to complete their transaction. One thing they noticed is that once a donor entered the shopping cart, there wasn’t much value proposition copy to motivate them through the transaction. They had seen tests that showed that a value proposition “sticky banner” in the process increased conversion rates for donations—even on high-conversion-rate pages. They planned to test different value proposition messages, but also wanted to aggregate the data to determine if the presence of a banner with any message increased conversion.
Research Question
Will a “sticky banner” with any value proposition outperform a control with no banner?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 36.0% | ||
| T1: | Treatment #1 | 41.0% | 13.7% | 95.5% |
This experiment has a required sample size of 760 in order to be valid. Since the experiment had a total sample size of 2,654, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 13.7% increase in conversion rate
× 0% increase in average gift
Key Learnings
A banner with any message increased conversion by 13.7%. Most of these increases were realized on desktop, though mobile had no decreases and no banner message had less than a 10% lift. This confirmed Compassion’s hypothesis that adding value proposition to the checkout process could increase motivation to complete the three-step process. Next, they looked at a different dataset to determine which messages drove the highest increases.