Alliance Defending Freedom

How supporting copy length on a Facebook ad affects email acquisition rate

Experiment ID: #8067

Alliance Defending Freedom

Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.

Experiment Summary

Timeframe: 12/05/2017 - 12/12/2017

Oral arguments for Jack Phillips–a client of Alliance Defending Freedom–began on December 5th. As the news covered this national story, Alliance Defending Freedom decided to launch a Facebook ad campaign urging people to sign a letter of support for Jack. Since they knew some people were unfamiliar with Jack, they wondered: Will increasing the length of supporting ad copy on a facebook ad increase clickthrough rate? To test this, they created a longer treatment and launched an A/B test to find out.

Research Question

Will increasing the length of the supporting ad copy increase clickthrough rate on a facebook ad?

Design

C: Control
T1: Treatment 1

Results

 Treatment NameClick RateRelative DifferenceConfidence
C: Control 1.3%
T1: Treatment 1 1.4%5.6% 89.3%

This experiment has a required sample size of 190,677 in order to be valid. Since the experiment had a total sample size of 258,517, and the level of confidence is not above 95% the experiment results are not valid.

Key Learnings

Alliance Defending Freedom that there was no statistical difference in email acquisition between the control and treatment ad. As we look at this test, it’s important to consider the timing of this campaign. Due to the relevant and captivating nature of this material, this campaign experienced a well above-average clickthrough and email acquisition rate. It’s likely that this played a factor in the A/B test (i.e. people were already pretty motivated to clickthrough, so there was less room for incremental optimization).


Experiment Documented by NextAfter

Question about experiment #8067

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.