How a collapsed banner on a mobile device impacts conversion Experiment ID: #9251

Focus on the Family

Focus on the Family is a global Christian ministry dedicated to helping families thrive. We provide help and resources for couples to build healthy marriages that reflect God's design, and for parents to raise their children according to morals and values grounded in biblical principles.

Experiment Summary

Timeframe: 6/26/2018 - 7/10/2018

Focus on the Family had a high urgency campaign that gave people the opportunity to have twice the impact of their gift through a match. In analyzing their campaign revenue from this year to last year, there was a 35% decrease in revenue. They hypothsized that the decrease in revenue might be, in part, due to the campaign banner at the top of their website. On mobile devices, specifically, the campaign was initially launched with the banner collapsed. But historically, an expanded version of the banner is what they have used in the past. In an effort to better understand if the collapsed banner was a contributing factor to the decrease in revenue, they proposed testing the expanded banner against the collapsed banner on mobile devices.

Research Question

How would the collapsed banner impact conversion?

Design

C: Expanded Banner
T1: Collapsed Banner

Results

Treatment Name Click Rate Relative Difference Confidence
C: Expanded Banner 0.16%
T1: Collapsed Banner 0.08% -47.2% 100.0%

This experiment has a required sample size of 16,993 in order to be valid. Since the experiment had a total sample size of 504,067, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    47.2% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

We weren’t able to validate the experiment off of donations, but we were able to validate it off of clicks to the donation page. The collapsed banner decreased clicks by 47%. This significant decrease in the number of people getting to the donation page can impact the number of people converting and making a gift. While we did see less donations from the collapsed banner, the total number of donations being made from the banner (expanded and collapsed) was really low. While the collapsed version did decrease the number of people getting to the donation page and had fewer gifts attached to it, there’s a different element impacting the decrease in revenue overall.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Courtney Gaines

Courtney is the Senior Director of Optimization at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.