How adding a personal story impacted email engagement
Alliance Defending Freedom
Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.
Experiment Summary
Timeframe: 07/12/2019 - 07/16/2019
Alliance Defending Freedom was continuing the longitudinal experiment to increase cultivation to a targeted segment of their email house file. The goal of these emails was to simply engage the audience and provide them with ongoing content and stewardship. With that in mind, we wanted to find ways to increase that engagement.
One hypothesis we developed was that we could potentially increase the personal-feel of these emails by including stories from the sender’s own life. We believed this personal touch might increase the likelihood that someone would click on the email.
Research Question
Will a personal story included in the email increase subscriber engagement?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 1.3% | ||
| T1: | Personal Story | 0.86% | -36.2% | 100.0% |
This experiment has a required sample size of 3,527 in order to be valid. Since the experiment had a total sample size of 61,123, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
36.2% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
Surprisingly, the personal story added to the beginning and end of the cultivation email resulted in a 36% decline in click through rate. After reading through the emails with a fresh perspective, our hypothesis for this decrease is that adding the personal story shifted the content of the email from subscriber-focused to sender-focused. The control starts out the email asking the subscribers a question while the treatment starts out talking about the sender (Sarah) themselves.
This is a good reminder that we need to craft every email with the goal of keeping the focus on the benefit to the subscriber.