How less teaser information in the weekly newsletter impacted clickthrough rates Experiment ID: #21161

The Fund for American Studies

Experiment Summary

Ended On: 5/21/2020

As a part of continued learning and testing, The Fund for American Studies was looking to further experiment with increasing the engagement (opens and clicks) from the weekly email newsletter.

After recently successfully testing their way into a simplified newsletter template (that removed all traces of marketing veneer), they decided to experiment with decreasing the amount of “teaser” copy that corresponded with each article.

Research Question

Will removing the teaser copy in the newsletter increase newsletter clickthrough rates?

Design

C: More Information
T1: Less Information

Results

Treatment Name Click Rate Relative Difference Confidence
C: More Information 4.3%
T1: Less Information 5.9% 39.1% 99.9%

This experiment has a required sample size of 1,339 in order to be valid. Since the experiment had a total sample size of 8,328, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    39.1% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

With 99.9% level of confidence, we observed an increase in newsletter ad clickthrough rates of 39.1%!

This experiment was run successively over a three-weekly newsletter send schedule, where we allowed the email system to randomize who received the simplified version from week to week.

The hypothesis is that there was an increase in article clicks because the corresponding “teaser text” that existed on the control allowed the email recipient to simply “skim” the article and determine either (a) whether or not the story interested them, or (b) whether or not they felt the teaser gave them enough information and there was no longer a need to click.

Where possible, consider removing any teaser text that corresponds to emails looking to generate click activity in newsletter format especially. The hyperlinked article headline should be enough to generate the intrigue you need to create the momentum needed to produce a higher volume of clicks.


Experiment Documented by...

Greg Colunga

Greg is Executive Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.