How a relevant subject line affected open rates for a newsletter | NextAfter
Dallas Theological Seminary

How a relevant subject line affected open rates for a newsletter

Experiment ID: #58997

Dallas Theological Seminary

The DTS mission is, “to glorify God by equipping godly servant-leaders for the proclamation of His Word and the building up of the body of Christ worldwide.” They strive to help men and women fulfill the Great Commission and the Great Commandment, or more simply: Teach Truth. Love Well.

Experiment Summary

Timeframe: 04/30/2021 - 05/03/2021

DTS introduced a new sender for their monthly newsletter and wanted to test how the file would respond to different subject lines. We set up a highly personalized control “Kevin, I wanted to encourage you today” and a highly relevant yet mysterious treatment “Happy Friday!” Neither the control nor the treatment gives away the contents of the email, but both subject lines indicate the message is both authentic from the sender and specifically for you. The treatment adds relevancy by including a day of the week and we wanted to see how this influenced email open rate so we ran a full A/B split test on the monthly newsletter send.

Research Question

We believe that adding relevancy for a housefile newsletter will achieve increased email engagment.

Design

C: Russell, I wanted to encourage you today
T1: Happy Friday!

Results

  Treatment Name Open Rate Relative Difference Confidence
C: Russell, I wanted to encourage you today 48.0%
T1: Happy Friday! 48.6% 1.3% 98.7%

This experiment has a required sample size of 51,002 in order to be valid. Since the experiment had a total sample size of 164,803, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    1.3% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

What we found in our A/B test was that the “Happy Friday” subject line did slightly increase the open rate by 1.3% with 98.7% confidence. However, we also analyzed the click rate of the newsletter, which experienced a 13.9% decrease with a 100% level of confidence. Taking this into consideration, moving forward we will continue to use personalization and mystery in subject lines to maintain and improve overall email engagement. We want the monthly newsletter recipients to not just open the email but to also engage with it and receive value from the contents included. Seeing the small bump in open rate is not nearly as valuable when we simultaneously saw such a decline on the treatment clickthrough rate. Looking ahead, we’d rather have the file be more deeply engaged in a way that they are taking that next step to demonstrate they are interested, connected and invested than to have a 1.3% improvement on the open rate for future newsletters. This is why it’s important to fully measure our results.


Experiment Documented by Rebekah Josefy
Rebekah Josefy is an Optimization Director at NextAfter.

Question about experiment #58997

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.