How a detailed subject line performs against a less detailed subject line Experiment ID: #10926

Leadership Institute

Experiment Summary

Timeframe: 5/30/2016 - 6/13/2016

The Leadership Institute sent out a Memorial Day email, and wanted to know if using subject lines that talked about or alluded to Memorial Day would perform better than a brief subject line that doesn’t let the reader know what the content is about.

Research Question

Do people want to know what’s in the email before they open?

Design

C: Remembering on this Memorial Day
T1: One moment
T2: Training and sacrifice

Results

Treatment Name Open Rate Relative Difference Confidence
C: Remembering on this Memorial Day 14.4%
T1: One moment 18.1% 26.2% 100.0%
T2: Training and sacrifice 16.9% 17.4% 100.0%

This experiment has a required sample size of 1,115 in order to be valid. Since the experiment had a total sample size of 32,784, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    26.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

While both treatments improved the open rate, the first treatment did much better, with an 18.1% open rate compared to the 16.9% open rate in the second treatment subject line of “Training and sacrifice” and a 14.4% open rate with the control of “Remembering on this Memorial Day”. This resulted in a 26.2% increase in traffic. The “One moment” subject line may resonate with the readers because it indicates that it should not take long to read the email. In the future, indicating that reading the email will not take too much time should be further tested against other successful subject lines to determine its efficacy in relation to those subject lines. Control had 27 clicks and a .25% click rate; Treatment 1 had 16 clicks and a .15% click rate; Treatment 2 had 9 clicks and a .08% click rate.


Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.