How removing the marketing template and images from a newsletter impacted open rate Experiment ID: #21544

Alley Cat Allies

Experiment Summary

Ended On: 6/25/2020

Alley Cat Allies was looking to experiment with whether or not they could increase their open or clickthrough rates on their monthly newsletter email. In reviewing, we determined that we would try to experiment with the email template.

Research Question

Can we increase open or clickthrough rates on the monthly newsletter by removing images and the marketing template?

Design

C: Styled Template
T1: Outlook Style Template

Results

Treatment Name Open Rate Relative Difference Confidence
C: Styled Template 16.7%
T1: Outlook Style Template 17.6% 5.3% 100.0%

This experiment has a required sample size of 13,988 in order to be valid. Since the experiment had a total sample size of 116,011, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    5.3% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

We observed a +5.3% increase in open rate with a 100% level of confidence from this experiment.

Regarding the clickthrough rate, we observed a -0.8% difference in clickthrough rate (LoC: 12%), which essentially meant there was no difference in the number of people clicking the email newsletter in the “outlook style” format (the treatment).

Simply put, the style of the newsletter template, when simplified, received a slightly better open rate, and no real difference in the number of people who would click the article links provided.

When looking at optimizing the program, this is actually a major victory for the Alley Cat Allies team. Why is that? Because the time it takes to produce the newsletter each month will come down tremendously with this new “outlook style” (or plain text style) of newsletter moving forward.

Moving forward, we should continue experimenting with other email types that use stylized templates to validate what impact (both in terms of opens, clicks or outcomes, but also in terms of “return on effort” to produce the newsletter/email each month by the staff) moving forward. Additionally, we recommend simplifying your newsletter and making all future newsletters run with the “outlook style” template moving forward.


Experiment Documented by...

Greg Colunga

Greg is Executive Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.