How the email presentation impacted click through rate
Live Action
Experiment Summary
Timeframe: 12/08/2016 - 12/09/2016
As part of a monthly appeal, Live Action wanted to test out the standard email design that had been in use for the last several months. This email template included the standard Live Action branding and an image of the Live Action president Lila Rose. The hypothesis was that if we could make the email feel more genuine, we could engage a greater proportion of the audience in the appeal.
To test this hypothesis, we stripped out the majority of the design elements and also softened the copy so that it did not have as many of the popular direct mail tactics such as bolding significant portions of text and including an ask array in the email (i.e. “would you consider making a $25, $50, or $100 gift.”).
Research Question
Will a simplified version of the email increase the number of subscribers getting to the page?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Designed Email | 1.3% | ||
| T1: | Simplified Design | 4.1% | 208.8% | 100.0% |
This experiment has a required sample size of 263 in order to be valid. Since the experiment had a total sample size of 14,218, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
208.8% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The simplified design increased the subscriber click through rate by 208%. This increased traffic resulted in a 50% lift to the number of donations made to to the treatment.
This experiment demonstrated the importance of facilitating a one-to-one relationship in mass-marketing emails. It is our belief that people give to people and when they feel marketed to, they are not as likely to engage with an organization. By making this email look more genuine, more people engaged in the message. As we plan for future iterations of testing, this is a concept that we want to continue to explore with further testing.