How changing the subject line of Evangelize emails to a name of a story increases opens Experiment ID: #17880

The Missionary Oblates of Mary Immaculate

The Missionary Oblates of Mary Immaculate are a Roman Catholic congregation of priests and brothers founded after the French Revolution by St. Eugene De Mazenod to work among the poor. Today there are nearly 4,000 missionaries working in more than 60 countries around the world.

Experiment Summary

Timeframe: 8/6/2019 - 8/7/2019

The monthly newsletter sent out by MAMI uses a standard subject line each month that clearly identifies the publication and the date it is sent. However, there was the hypothesis that if we made the subject line more casual, we could increase the open rate. We decided to test this with a 50/50 split.

Research Question

Will a more casual subject line increase email engagement?


C: Evangelize Newsletter - Subject Line
T1: It's Great to be Back! - Subject Line


Treatment Name Open Rate Relative Difference Confidence
C: Evangelize Newsletter - Subject Line 34.2%
T1: It's Great to be Back! - Subject Line 39.3% 14.7% 100.0%

This experiment has a required sample size of 704 in order to be valid. Since the experiment had a total sample size of 44,480, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    14.7% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

We observed a statistically significant 14.7% lift to open rate by using a more casual subject line. The tricky aspect of this finding is that it may be a one-time impact that would fade over time. In order to further validate this finding, we would want to continue to test this tactic over the next several months.

Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.