How case-specific value proposition copy impacts email acquisition
Alliance Defending Freedom
Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.
Experiment Summary
Timeframe: 09/13/2017 - 10/17/2017
Alliance Defending Freedom recently conducted a test with their exit intent offer, asking: Will a more case-specific offer increase email acquisition? The results of that test were inconclusive but prompted a similar test on their ‘Jack Phillips’ information and email acquisition page. They asked: Will a case-specific newsletter offer increase email acquisition? They created two versions and launched an A/B test to find out.
Research Question
Will an newsletter signup offer geared towards a particular legal case increase email acquisition?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 1.7% | ||
| T1: | Treatment 1 | 2.2% | 25.7% | 71.4% |
This experiment has a required sample size of 7,427 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
Again, Alliance Defending Freedom found that the results were inconclusive, since the test did not reach a statistical significance of 95% and a statistical power of 80%. We would naturally believe that aligning the newsletter offer with the related information on the page would increase email acquisition. And perhaps, it can if given enough time and traffic. However, this test cannot confirm that hypothesis, and thus, requires further testing to prove.