How adding a personal story impacted email engagement Experiment ID: #17004

Alliance Defending Freedom

Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.

Experiment Summary

Timeframe: 7/12/2019 - 7/16/2019

Alliance Defending Freedom was continuing the longitudinal experiment to increase cultivation to a targeted segment of their email house file. The goal of these emails was to simply engage the audience and provide them with ongoing content and stewardship. With that in mind, we wanted to find ways to increase that engagement.

One hypothesis we developed was that we could potentially increase the personal-feel of these emails by including stories from the sender’s own life. We believed this personal touch might increase the likelihood that someone would click on the email.

Research Question

Will a personal story included in the email increase subscriber engagement?


C: Control
T1: Personal Story


Treatment Name Click Rate Relative Difference Confidence
C: Control 1.3%
T1: Personal Story 0.86% -36.2% 100.0%

This experiment has a required sample size of 3,527 in order to be valid. Since the experiment had a total sample size of 61,123, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    36.2% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

Surprisingly, the personal story added to the beginning and end of the cultivation email resulted in a 36% decline in click through rate. After reading through the emails with a fresh perspective, our hypothesis for this decrease is that adding the personal story shifted the content of the email from subscriber-focused to sender-focused. The control starts out the email asking the subscribers a question while the treatment starts out talking about the sender (Sarah) themselves.

This is a good reminder that we need to craft every email with the goal of keeping the focus on the benefit to the subscriber.

Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.