How removing the marketing template and images from a newsletter impacted open rate
Alley Cat Allies
Experiment Summary
Ended On: 06/25/2020
Alley Cat Allies was looking to experiment with whether or not they could increase their open or clickthrough rates on their monthly newsletter email. In reviewing, we determined that we would try to experiment with the email template.
Research Question
Can we increase open or clickthrough rates on the monthly newsletter by removing images and the marketing template?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Styled Template | 16.7% | ||
| T1: | Outlook Style Template | 17.6% | 5.3% | 100.0% |
This experiment has a required sample size of 13,988 in order to be valid. Since the experiment had a total sample size of 116,011, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
5.3% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
We observed a +5.3% increase in open rate with a 100% level of confidence from this experiment.
Regarding the clickthrough rate, we observed a -0.8% difference in clickthrough rate (LoC: 12%), which essentially meant there was no difference in the number of people clicking the email newsletter in the “outlook style” format (the treatment).
Simply put, the style of the newsletter template, when simplified, received a slightly better open rate, and no real difference in the number of people who would click the article links provided.
When looking at optimizing the program, this is actually a major victory for the Alley Cat Allies team. Why is that? Because the time it takes to produce the newsletter each month will come down tremendously with this new “outlook style” (or plain text style) of newsletter moving forward.
Moving forward, we should continue experimenting with other email types that use stylized templates to validate what impact (both in terms of opens, clicks or outcomes, but also in terms of “return on effort” to produce the newsletter/email each month by the staff) moving forward. Additionally, we recommend simplifying your newsletter and making all future newsletters run with the “outlook style” template moving forward.