The Heritage Foundation

How increasing the size of the text on an offer landing page did not impact the email registration rate

Experiment ID: #14307

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Timeframe: 04/12/2019 - 04/15/2019

When The Heritage Foundation launched a new offer, we were not achieving the email registration rate that we would normally see. So, in an attempt to boost the email registration rate, we launched a simple experiment to boost the size of the text across both desktop and mobile versions to see if readability was prohibiting us from achieving the results we wanted.

Research Question

Will improved readability increase the email registration rate for the Religious Freedom offer?

Design

C: control - smaller text
T1: treatment - larger text

Results

 Treatment NameConv. RateRelative DifferenceConfidence
C: control - smaller text 28.6%
T1: treatment - larger text 26.2%-8.3% 80.9%

This experiment has a required sample size of 2,691 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

With an 80.9% level of confidence, we found that the large text actually decreased the email registration rate by -8.3%.

We will now turn our testing towards the message itself, having eliminated something structural with the page as a variable that is limiting our effectiveness.


Experiment Documented by Greg Colunga
Greg Colunga is Executive Vice President at NextAfter.

Question about experiment #14307

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.