Luther Seminary

How a timely and relevant subject line impacted clicks on a cultivation email

Experiment ID: #164609

Luther Seminary

Experiment Summary

Timeframe: 11/18/2023 - 11/20/2023

When sending a cultivation email to the housefile, we wanted to see if we could increase clicks by calling attention in the subject line that the content of the message was seasonal and relevant. The control subject line drove curiosity “embracing the unpredictable” but we wondered if more people would engage with the email if we let them know it couldn’t wait to be opened or they would lose out on the timeliness of the article inside.

Research Question

We believe that increasing the relevancy of a subject line for cultivation email recipients will achieve an increase in email engagement because the timeliness of the message is increased.

Design

C: Embracing the unpredictable
T1: A must-read before Thanksgiving!

Results

 Treatment NameClick RateRelative DifferenceConfidence
C: Embracing the unpredictable 2.8%
T1: A must-read before Thanksgiving! 5.0%76.4% 100.0%

This experiment has a required sample size of 621 in order to be valid. Since the experiment had a total sample size of 28,556, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    76.4% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The key learning from this experiment is that increasing the relevancy of the subject line for cultivation email recipients resulted in a significant increase in email engagement, specifically the Clicks metric. The treatment subject line, which highlighted the timeliness of the article inside, outperformed the control subject line in terms of driving clicks and led to a 76% increase in clickthrough rate with 100% level of confidence.

In future experimentation, it would be useful to explore subject lines that strike a balance between timeliness and curiosity. For example, a subject line that hints at seasonal and relevant content while also creating a sense of curiosity or intrigue might be effective in driving both opens and clicks.

Overall, this experiment highlights the importance of continuously testing and optimizing subject lines to maximize email engagement. By using data-driven insights, we can better understand what drives recipient behavior and tailor our messaging accordingly.


Experiment Documented by Rebekah Josefy
Rebekah Josefy is an Optimization Director at NextAfter.

Question about experiment #164609

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.