How a collapsed banner on a mobile device impacts conversion
Focus on the Family
Focus on the Family is a global Christian ministry dedicated to helping families thrive. We provide help and resources for couples to build healthy marriages that reflect God's design, and for parents to raise their children according to morals and values grounded in biblical principles.
Experiment Summary
Timeframe: 06/26/2018 - 07/10/2018
Focus on the Family had a high urgency campaign that gave people the opportunity to have twice the impact of their gift through a match. In analyzing their campaign revenue from this year to last year, there was a 35% decrease in revenue. They hypothsized that the decrease in revenue might be, in part, due to the campaign banner at the top of their website. On mobile devices, specifically, the campaign was initially launched with the banner collapsed. But historically, an expanded version of the banner is what they have used in the past. In an effort to better understand if the collapsed banner was a contributing factor to the decrease in revenue, they proposed testing the expanded banner against the collapsed banner on mobile devices.
Research Question
How would the collapsed banner impact conversion?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Expanded Banner | 0.16% | ||
| T1: | Collapsed Banner | 0.08% | -47.2% | 100.0% |
This experiment has a required sample size of 16,993 in order to be valid. Since the experiment had a total sample size of 504,067, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
47.2% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
We weren’t able to validate the experiment off of donations, but we were able to validate it off of clicks to the donation page. The collapsed banner decreased clicks by 47%. This significant decrease in the number of people getting to the donation page can impact the number of people converting and making a gift. While we did see less donations from the collapsed banner, the total number of donations being made from the banner (expanded and collapsed) was really low. While the collapsed version did decrease the number of people getting to the donation page and had fewer gifts attached to it, there’s a different element impacting the decrease in revenue overall.