How explaining the content inside an email newsletter affects email acquisition rate
Leadership Institute
Experiment Summary
Ended On: 10/23/2019
Leadership Institute’s website CampusReform.org had a sticky bar that showed for mobile devices and gave them the opportunity to sign up for their email list. The team wanted to test different options to see if one copy variant could increase conversion rate. The control had the headline “Don’t miss a story”, and offered CampusWire, the weekly newsletter. They decided to test two different angles. First, they moved the CampusWire name to the headline to see if introducing that earlier could pique curiosity about the newsletter and inspire people to sign up. In doing this, they realized that repeating the name in the short paragraph below the headline would be redundant, so they inserted new copy about leftist bias, in order to fill the space and continue to motivate the reader. Then, they created an unbranded, yet more specific headline in the style of the control: “Don’t miss a campus report”. They altered the paragraph beneath the headline to further explain the concept of a “campus report”, showing that it comes straight from student reporters and includes stories of leftist bias. They launched an A/B/C test to find a winner.
Research Question
How will branded or unbranded specificity affect email acquisition rate?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 5.1% | ||
| T1: | Treatment #1 | 5.1% | 0.4% | 17.1% |
| T2: | Treatment #2 | 5.8% | 14.0% | 100.0% |
This experiment has a required sample size of 11,489 in order to be valid. Since the experiment had a total sample size of 353,651, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 14.0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The first (branded) treatment showed no increase in conversion. However, the second treatment, which led with the “campus report” language and then included a deeper description of the content, did increase conversion by 14%.
This test is a good reminder that email newsletters are a commodity—and that naming a relatively unknown brand doesn’t necessarily strengthen the value proposition.
It is also worth noting that the other major difference between the winning ad and the control was with regards to the frequency of the newsletter. The winner promoted a daily newsletter and the control was weekly. We’ll need to isolate this variable to know if this was the determining factor in the winning treatment.