How a plain, stripped down, email impacts engagement Experiment ID: #19286

Back to the Bible

Experiment Summary

Ended On: 12/9/2019

Back to the Bible offered people on their emailing file the opportunity to sign-up for a free Bible Reading Plan. The control email that was being sent used a marketing template and lots of images. We hypothesized that a stripped down email could help improve email engagement.

Research Question

Would a stripped down email help improve email engagement?

Design

C: Control
T1: Plain Text

Results

Treatment Name Click Rate Relative Difference Confidence
C: Control 0.82%
T1: Plain Text 0.97% 19.4% 98.9%

This experiment has a required sample size of 27,337 in order to be valid. Since the experiment had a total sample size of 91,257, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    19.4% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

By sending a plain text email, we were able to increase email engagement by 19%. Not only that, but we also saw an increase in opens – even though the subject lines were identical. This tells us that we had increased deliverability and inboxing. People appreciate hearing from Arnie – as a person. The tone and copy of the email was believable that he actually sent the email to the file. By stripping the email of the marketing elements (images, template, etc) we were able to reinforce this sentiment as well.


Experiment Documented by...

Courtney Gaines

Courtney is a Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.