How a relevant subject line affected open rates for a newsletter
Dallas Theological Seminary
The DTS mission is, “to glorify God by equipping godly servant-leaders for the proclamation of His Word and the building up of the body of Christ worldwide.” They strive to help men and women fulfill the Great Commission and the Great Commandment, or more simply: Teach Truth. Love Well.
Experiment Summary
Timeframe: 04/30/2021 - 05/03/2021
DTS introduced a new sender for their monthly newsletter and wanted to test how the file would respond to different subject lines. We set up a highly personalized control “Kevin, I wanted to encourage you today” and a highly relevant yet mysterious treatment “Happy Friday!” Neither the control nor the treatment gives away the contents of the email, but both subject lines indicate the message is both authentic from the sender and specifically for you. The treatment adds relevancy by including a day of the week and we wanted to see how this influenced email open rate so we ran a full A/B split test on the monthly newsletter send.
Research Question
We believe that adding relevancy for a housefile newsletter will achieve increased email engagment.
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Russell, I wanted to encourage you today | 48.0% | ||
| T1: | Happy Friday! | 48.6% | 1.3% | 98.7% |
This experiment has a required sample size of 51,002 in order to be valid. Since the experiment had a total sample size of 164,803, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
1.3% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
What we found in our A/B test was that the “Happy Friday” subject line did slightly increase the open rate by 1.3% with 98.7% confidence. However, we also analyzed the click rate of the newsletter, which experienced a 13.9% decrease with a 100% level of confidence. Taking this into consideration, moving forward we will continue to use personalization and mystery in subject lines to maintain and improve overall email engagement. We want the monthly newsletter recipients to not just open the email but to also engage with it and receive value from the contents included. Seeing the small bump in open rate is not nearly as valuable when we simultaneously saw such a decline on the treatment clickthrough rate. Looking ahead, we’d rather have the file be more deeply engaged in a way that they are taking that next step to demonstrate they are interested, connected and invested than to have a 1.3% improvement on the open rate for future newsletters. This is why it’s important to fully measure our results.