How adding mystery and shortening the subject line improves opens
The Missionary Oblates of Mary Immaculate
The Missionary Oblates of Mary Immaculate are a Roman Catholic congregation of priests and brothers founded after the French Revolution by St. Eugene De Mazenod to work among the poor. Today there are nearly 4,000 missionaries working in more than 60 countries around the world.
Experiment Summary
Ended On: 10/09/2019
This was tested in Mailchimp. We noticed the subject line was to long on some devices and would cut off half of the text. We also wanted to add mystery to the subject line.
Research Question
We wanted to see if adding mystery to the subject line and shortening the subject to fit on all devices would improve opens.
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Our Lady of Fatima "The Miracle of the Sun" | 31.2% | ||
| T1: | The Miracle of the Sun | 30.2% | -3.0% | 99.0% |
This experiment has a required sample size of 18,291 in order to be valid. Since the experiment had a total sample size of 63,676, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
3.0% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
Our hypothesis was off and it seems that adding the mystery and shortening the subject did not work for this email. It could be because of people being very interested in “Our Lady of Fatima” and from seeing that they were interested to see what was in the copy. Moving forward we will have Our Lady of Fatima in the subject line and test other subject lines that has “Our Lady” in it to see if that’s what is increasing opens.