How removing a newsletter’s marketing template impacted clickthrough rate Experiment ID: #20978
The Fund for American Studies
Ended On: 5/1/2020
The Fund for American Studies (TFAS) was looking to increase email engagement (open and clickthrough rates) for newsletter recipients, and in doing so, decided to turn towards experimentation to iteratively increase results.
The first test was to see whether removing the email marketing template for the newsletter would increase open or clickthrough rates.
By removing the stylized marketing template for the newsletter, can we improve newsletter open or clickthrough rates?
|Treatment Name||Click Rate||Relative Difference||Confidence|
|C:||Control - Newsletter w/Marketing Template||0.44%|
|T1:||Treatment - Newsletter w/out Marketing Template||0.75%||68.5%||100.0%|
This experiment has a required sample size of 4,924 in order to be valid. Since the experiment had a total sample size of 36,927, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
68.5% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
With 99.9% level of confidence, we observed a 68.5% increase in newsletter article clickthrough rates.
We also observed with a 100% level of confidence an increase in open rates by +35.1%.
This was experimented across two successive newsletters to verify that there wasn’t a one-off result. Both tests were consistent in their lift across both metrics, and the numbers shown are aggregated totals across both email tests.
This further proves that by stripping out the newsletter template to appear more “plain text” in nature, that we can inbox at a better rate. By inboxing more newsletters, we can reach more people (which creates more opens) and because of more openers/readers of the email, we can generate a higher volume of clicks.
Next Steps: Further experimentation on increasing engagement and article consumption or responses to the sender are advised.