How multiple links impacted subscriber engagement Experiment ID: #16124

Alliance Defending Freedom

Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.

Experiment Summary

Timeframe: 5/24/2019 - 5/28/2019

As part of the weekly cultivation efforts with the targeted group of subscribers, we wanted to find new and creative way to increase the overall engagement of the group. Specifically, we wanted to get more of them to open the emails and read the content. One of the more simple solutions was to potentially increase the amount of content we talk about in the email. Our hypothesis was that we can add additional related links to the email and get more people to engage with some content.

Research Question

Does adding additional links to the email increase the likelihood of a reader to visit the website?


C: Single Link
T1: Multiple Links


Treatment Name Click Rate Relative Difference Confidence
C: Single Link 3.9%
T1: Multiple Links 4.8% 23.2% 97.8%

This experiment has a required sample size of 3,983 in order to be valid. Since the experiment had a total sample size of 11,079, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    23.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

We saw a 23% increase in overall email clicks by including an additional article link in the email. However, when analyzing the email, we noticed that there was a disparity in the segment open rates that was significantly larger than it should have been. Both emails had the same sender, send time, subject line, and preview text which should have led to similar open rates. When we dug in deeper, we found that one of the segments had significantly more gmail addresses which had deliverability issues with this email send. As a result, we will want to retest this concept in a future email with a better distributed sample set.

Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.