How removing a newsletter’s marketing template impacted clickthrough rate Experiment ID: #20949

Americans for Prosperity

Experiment Summary

Ended On: 4/30/2020

The Fund for American Studies (TFAS) was looking to increase email engagement (open and clickthrough rates) for newsletter recipients, and in doing so, decided to turn towards experimentation to iteratively increase results.

The first test was to see whether removing the email marketing template for the newsletter would increase open or clickthrough rates.

Research Question

By removing the stylized marketing template for the newsletter, can we improve newsletter open or clickthrough rates?

Design

C: Control - Newsletter w/Marketing Template
T1: Treatment - Newsletter w/out Marketing Template

Results

Treatment Name Click Rate Relative Difference Confidence
C: Control - Newsletter w/Marketing Template 0.44%
T1: Treatment - Newsletter w/out Marketing Template 0.75% 68.5% 100.0%

This experiment has a required sample size of 4,924 in order to be valid. Since the experiment had a total sample size of 36,927, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    68.5% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

With 99.9% level of confidence, we observed a 68.5% increase in newsletter article clickthrough rates.

We also observed with a 100% level of confidence an increase in open rates by +35.1%.

This was experimented across two successive newsletters to verify that there wasn’t a one-off result. Both tests were consistent in their lift across both metrics, and the numbers shown are aggregated totals across both email tests.

Key Takeaway:

This further proves that by stripping out the newsletter template to appear more “plain text” in nature, that we can inbox at a better rate. By inboxing more newsletters, we can reach more people (which creates more opens) and because of more openers/readers of the email, we can generate a higher volume of clicks.

Next Steps: Further experimentation on increasing engagement and article consumption or responses to the sender are advised.


Experiment Documented by...

Greg Colunga

Greg is Executive Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.