How adding the word “Free” increased open rates Experiment ID: #13854

The Missionary Oblates of Mary Immaculate

The Missionary Oblates of Mary Immaculate are a Roman Catholic congregation of priests and brothers founded after the French Revolution by St. Eugene De Mazenod to work among the poor. Today there are nearly 4,000 missionaries working in more than 60 countries around the world.

Experiment Summary

Ended On: 4/5/2019

In past novena emails we used the subject “Join in an Easter Novena”. We wanted to see if we could increase the amount opens by including the word “Free”

Research Question

We wanted to see if the word “Free” would increase open rates.


C: Join in an Easter Novena
T1: A Free Easter Novena for You


Treatment Name Open Rate Relative Difference Confidence
C: Join in an Easter Novena 32.4%
T1: A Free Easter Novena for You 33.6% 3.7% 99.6%

This experiment has a required sample size of 11,865 in order to be valid. Since the experiment had a total sample size of 51,466, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    3.7% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

We saw a 3.7% increase in traffic from adding the word “Free” to our subject line. Another big takeaway from this is that for the Control there was a 26.9% ctr. With the Treatment 29.5% in ctr. Giving the Treatment a 2.6% higher increase in clicks than the Control. The word “Free” could have given people the incentive to click through faster rather than having to read the copy in the email first.

Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.