How minimized navigation and design affects conversion
Compassion International
Experiment Summary
Timeframe: 02/04/2021 - 02/16/2021
Compassion International had found “birthday search” to be a compelling offer for new donor acquisition. For many years, they had leveraged a stripped down page to present this offer, with confetti and a headline that gave the visitor a reason to give. However, they had never tested this against a similar landing page experience that lived within the general Compassion design. They wanted to understand whether the old template, which was outside of current brand standards, was worth keeping up. They decided to test this through an A/B test.
Research Question
We believe that a stripped down page for birthday searchers will achieve an increase in sponsor conversion.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Original | 1.3% | ||
| T1: | Variation #1 | 0.84% | -36.9% | 98.8% |
This experiment has a required sample size of 3,431 in order to be valid. Since the experiment had a total sample size of 11,169, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 36.9% decrease in conversion rate
× 0% increase in average gift
Key Learnings
The treatment, with the stripped down navigation (which removed the Compassion “login” link), produced a 36.9% decrease in conversion. However, it’s important to understand exactly what happened here, as there were two different responses based on the device type.
First, mobile visitors had a 45.0% decrease in conversion – with a 100% level of confidence. Since most of the traffic was mobile, this was the deciding factor in the test. However, desktop traffic had a 235.8% increase in conversion – with a 94% level of confidence.
This shows that different designs can produce opposite results with the same page—which stresses the importance of responsive design. It also may hint that something was wrong with the mobile version of the treatment that might be corrected to produce a similar result.