How reducing decision friction impacted email clicks Experiment ID: #20762

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Ended On: 4/22/2020

As part of a weekly cultivation series to a segment of their housefile, The Heritage Foundation was sending an invitation that included several links to participate in upcoming webinars taking place on various dates and covering various topics. They wanted to test if sending a single link to an individual webinar would increase email engagement. Our hypothesis was that eliminating the additional webinar options and focusing the call to action on one specific event would optimize for more clicks and lead to additional traffic to the website.

Research Question

Will reducing decision friction by minimizing the number of links included in the email increase click rate?

Design

C: Control - Multiple Links
T1: Treatment - Individual Link

Results

Treatment Name Click Rate Relative Difference Confidence
C: Control - Multiple Links 0.33%
T1: Treatment - Individual Link 0.43% 31.5% 100.0%

This experiment has a required sample size of 27,186 in order to be valid. Since the experiment had a total sample size of 315,774, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    31.5% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

We saw a 31.5% increase in email clicks by removing the additional webinar links in the email. This indicates that we can drive more traffic to our website from email marketing when using a single call to action with a specific focus rather than presenting additional links to various content. To increase email engagement and website traffic, The Heritage Foundation should continue using this approach for future webinar invitations.


Experiment Documented by...

Rebekah Josefy

Rebekah is Optimization Associate. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.