How condensed and intriguing hyperlink copy impacted newsletter clickthrough rates Experiment ID: #21892

Alley Cat Allies

Experiment Summary

Ended On: 7/24/2020

As a part of iterative testing with the Alley Cat Allies newsletter in the name of increasing email engagement rates, we wanted to experiment with using shorter, more intriguing article hyperlinks within the monthly newsletter to determine what impact it might have moving forward.

Research Question

Will shorter and more intriguing link descriptions increase newsletter clickthrough rate?

Design

C: Control
T1: Article Teaser in Hyperlinks

Results

Treatment Name Click Rate Relative Difference Confidence
C: Control 0.92%
T1: Article Teaser in Hyperlinks 1.0% 13.2% 96.3%

This experiment has a required sample size of 50,892 in order to be valid. Since the experiment had a total sample size of 115,028, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    13.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

With a 96.2% level of confidence, we observed a +13.2% increase in clickthrough rate from the treatment.

The basic thesis of this experiment is that by giving away less of what happens “post-click” away from the articles being presented in the newsletter, and instead by creating “intrigue” for what happens after the reader clicks to learn more, we can drive better newsletter clickthrough rates moving forward.

Not only does this approach generate more engagement, but it is also an easier version of the monthly newsletter to create. This means that the team can spend more time focusing upon writing teasers for what is in the article and move on with other priorities to drive better newsletter engagement moving forward.


Experiment Documented by...

Greg Colunga

Greg is Executive Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.