How minimal content impacts subscriber engagement
Alliance Defending Freedom
Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.
Experiment Summary
Timeframe: 09/06/2019 - 09/10/2019
The primary goal of Alliance Defending Freedom’s weekly cultivation series is to engage subscribers with content that is central to ADF’s mission. With this in mind, past optimization efforts have centered around finding the optimal tactics to lift open and click through rates. All of the emails are sent from the same sender, “Sarah”, and are written to appear as one-to-one as possible.
In recent discussions, a hypothesis was proposed that we could potentially increase the number of visitors clicking by reducing the amount of content we put into the email. The theory was based upon the thought that we may be spending too much time “selling” the content instead of letting the visitors get to the information.
To test this hypothesis, we created a treatment version of the weekly email with a dramatically simplified message.
Research Question
Will a simplified message that focuses on the links rather than the reason to click increase overall clickthrough rate?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 7.1% | ||
| T1: | Minimal Content | 8.0% | 13.0% | 95.0% |
This experiment has a required sample size of 6,272 in order to be valid. Since the experiment had a total sample size of 12,604, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
13.0% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
We found that we were able to increase overall click through rate by 13% by reducing the amount of effort we spent selling the content in the email. If this proves to be a universal tactic, this could significantly help speed up the content creation process. However, this concept will need to be further tested to ensure it wasn’t an anomaly.