How supporting copy length on a Facebook ad affects email acquisition rate
Alliance Defending Freedom
Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.
Experiment Summary
Timeframe: 12/05/2017 - 12/12/2017
Oral arguments for Jack Phillips–a client of Alliance Defending Freedom–began on December 5th. As the news covered this national story, Alliance Defending Freedom decided to launch a Facebook ad campaign urging people to sign a letter of support for Jack. Since they knew some people were unfamiliar with Jack, they wondered: Will increasing the length of supporting ad copy on a facebook ad increase clickthrough rate? To test this, they created a longer treatment and launched an A/B test to find out.
Research Question
Will increasing the length of the supporting ad copy increase clickthrough rate on a facebook ad?
Design
Results
Treatment Name | Click Rate | Relative Difference | Confidence | |
---|---|---|---|---|
C: | Control | 1.3% | ||
T1: | Treatment 1 | 1.4% | 5.6% | 89.3% |
This experiment has a required sample size of 190,677 in order to be valid. Since the experiment had a total sample size of 258,517, and the level of confidence is not above 95% the experiment results are not valid.
Key Learnings
Alliance Defending Freedom that there was no statistical difference in email acquisition between the control and treatment ad. As we look at this test, it’s important to consider the timing of this campaign. Due to the relevant and captivating nature of this material, this campaign experienced a well above-average clickthrough and email acquisition rate. It’s likely that this played a factor in the A/B test (i.e. people were already pretty motivated to clickthrough, so there was less room for incremental optimization).
Question about experiment #8067
If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.