How multiple links impacted subscriber engagement
Alliance Defending Freedom
Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.
Experiment Summary
Timeframe: 05/24/2019 - 05/28/2019
As part of the weekly cultivation efforts with the targeted group of subscribers, we wanted to find new and creative way to increase the overall engagement of the group. Specifically, we wanted to get more of them to open the emails and read the content. One of the more simple solutions was to potentially increase the amount of content we talk about in the email. Our hypothesis was that we can add additional related links to the email and get more people to engage with some content.
Research Question
Does adding additional links to the email increase the likelihood of a reader to visit the website?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Single Link | 3.9% | ||
| T1: | Multiple Links | 4.8% | 23.2% | 97.8% |
This experiment has a required sample size of 3,983 in order to be valid. Since the experiment had a total sample size of 11,079, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
23.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
We saw a 23% increase in overall email clicks by including an additional article link in the email. However, when analyzing the email, we noticed that there was a disparity in the segment open rates that was significantly larger than it should have been. Both emails had the same sender, send time, subject line, and preview text which should have led to similar open rates. When we dug in deeper, we found that one of the segments had significantly more gmail addresses which had deliverability issues with this email send. As a result, we will want to retest this concept in a future email with a better distributed sample set.