How changing the subject line of Evangelize emails to a name of a story increases opens
The Missionary Oblates of Mary Immaculate
The Missionary Oblates of Mary Immaculate are a Roman Catholic congregation of priests and brothers founded after the French Revolution by St. Eugene De Mazenod to work among the poor. Today there are nearly 4,000 missionaries working in more than 60 countries around the world.
Experiment Summary
Timeframe: 08/06/2019 - 08/07/2019
The monthly newsletter sent out by MAMI uses a standard subject line each month that clearly identifies the publication and the date it is sent. However, there was the hypothesis that if we made the subject line more casual, we could increase the open rate. We decided to test this with a 50/50 split.
Research Question
Will a more casual subject line increase email engagement?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Evangelize Newsletter - Subject Line | 34.2% | ||
| T1: | It's Great to be Back! - Subject Line | 39.3% | 14.7% | 100.0% |
This experiment has a required sample size of 704 in order to be valid. Since the experiment had a total sample size of 44,480, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
14.7% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
We observed a statistically significant 14.7% lift to open rate by using a more casual subject line. The tricky aspect of this finding is that it may be a one-time impact that would fade over time. In order to further validate this finding, we would want to continue to test this tactic over the next several months.