How less teaser information in the weekly newsletter impacted clickthrough rates Experiment ID: #21161
The Fund for American Studies
Ended On: 5/21/2020
As a part of continued learning and testing, The Fund for American Studies was looking to further experiment with increasing the engagement (opens and clicks) from the weekly email newsletter.
After recently successfully testing their way into a simplified newsletter template (that removed all traces of marketing veneer), they decided to experiment with decreasing the amount of “teaser” copy that corresponded with each article.
Will removing the teaser copy in the newsletter increase newsletter clickthrough rates?
|Treatment Name||Click Rate||Relative Difference||Confidence|
This experiment has a required sample size of 1,339 in order to be valid. Since the experiment had a total sample size of 8,328, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
39.1% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
With 99.9% level of confidence, we observed an increase in newsletter ad clickthrough rates of 39.1%!
This experiment was run successively over a three-weekly newsletter send schedule, where we allowed the email system to randomize who received the simplified version from week to week.
The hypothesis is that there was an increase in article clicks because the corresponding “teaser text” that existed on the control allowed the email recipient to simply “skim” the article and determine either (a) whether or not the story interested them, or (b) whether or not they felt the teaser gave them enough information and there was no longer a need to click.
Where possible, consider removing any teaser text that corresponds to emails looking to generate click activity in newsletter format especially. The hyperlinked article headline should be enough to generate the intrigue you need to create the momentum needed to produce a higher volume of clicks.