How creating a simpler email format increases donations and revenue Experiment ID: #21354

The Missionary Oblates of Mary Immaculate

The Missionary Oblates of Mary Immaculate are a Roman Catholic congregation of priests and brothers founded after the French Revolution by St. Eugene De Mazenod to work among the poor. Today there are nearly 4,000 missionaries working in more than 60 countries around the world.

Experiment Summary

Ended On: 6/9/2020

Missionary Oblates send out a monthly evangelize newsletter. They use the simpler format on all other emails that come from their organization as a way of creating a one-to-one relationship with the subscribers. They want to see if this same concept would apply to the stylized newsletter sent out each month. To do this, instead of breaking up all the stories, they put the entire first story into the email followed by the other links. The goal was the measure the impact on engagement (including giving).

Research Question

Would creating a simpler format for our Evangelize emails increase donations?

Design

C: Control
T1: Treatment #1

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Control 0.02%
T1: Treatment #1 0.05% 171.4% 98.1%

This experiment has a required sample size of 26,789 in order to be valid. Since the experiment had a total sample size of 77,274, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 171.4% increase in conversion rate
× 0% increase in average gift

Key Learnings

We ended up with a 171% increase in conversion rate and more than tripled our revenue! We learned that adding the story with a straight donation link increased donations significantly. The email was also more personal. Based off of these results we will continue using the simpler format with less links.


Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.