How a supporting image affects email acquisition rate on a landing page Experiment ID: #8146

Alliance Defending Freedom

Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.

Experiment Summary

Timeframe: 12/5/2017 - 12/19/2017

Alliance Defending Freedom recently launched a campaign to rally support around Jack Phillips–one of the clients they are defending at the Supreme Court. They found that adding additional copy to their facebook ads did not increase or decrease clickthrough rate. They wondered: Will removing the image of Jack Phillips on the email acquisition page increase the overall email acquisition rate? They created a treatment version (without the image of Jack) and launched an A/B test to find out.

Research Question

Will removing a supporting image increase email acquisition rate on a landing page?

Design

C: Control
T1: Treatment 1

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Control 71.7%
T1: Treatment 1 72.5% 1.1% 78.9%

This experiment has a required sample size of 23,859 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

Alliance Defending Freedom found that the the treatment version actually had no statistically significant effect on email acquisition. In other words, we saw no difference in conversion on the page that lacked the supporting image. This is an interesting learning for all of us to consider, as we build new campaign pages. This test suggests that the presence (or lack thereof) of a smaller image doesn’t have a dramatic impact on visitor motivation.

However, it’s also important to note that the conversion rate for this page was much higher than the average acquisition page (converting over 70% of all visitors). Motivation may have been so high, in fact, that it overpowered any page elements that would have otherwise affected page users. Thus, Alliance Defending Freedom will continue to test the image element in order to be certain of its impact.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.