How minimal content impacts subscriber engagement Experiment ID: #18149

Alliance Defending Freedom

Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.

Experiment Summary

Timeframe: 9/6/2019 - 9/10/2019

The primary goal of Alliance Defending Freedom’s weekly cultivation series is to engage subscribers with content that is central to ADF’s mission. With this in mind, past optimization efforts have centered around finding the optimal tactics to lift open and click through rates. All of the emails are sent from the same sender, “Sarah”, and are written to appear as one-to-one as possible.

In recent discussions, a hypothesis was proposed that we could potentially increase the number of visitors clicking by reducing the amount of content we put into the email. The theory was based upon the thought that we may be spending too much time “selling” the content instead of letting the visitors get to the information.

To test this hypothesis, we created a treatment version of the weekly email with a dramatically simplified message.

Research Question

Will a simplified message that focuses on the links rather than the reason to click increase overall clickthrough rate?


C: Control
T1: Minimal Content


Treatment Name Click Rate Relative Difference Confidence
C: Control 7.1%
T1: Minimal Content 8.0% 13.0% 95.0%

This experiment has a required sample size of 6,272 in order to be valid. Since the experiment had a total sample size of 12,604, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    13.0% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

We found that we were able to increase overall click through rate by 13% by reducing the amount of effort we spent selling the content in the email. If this proves to be a universal tactic, this could significantly help speed up the content creation process. However, this concept will need to be further tested to ensure it wasn’t an anomaly.

Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.