How larger font size decreased email acquisition rate for The Guide to the Constitution offer
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Ended On: 04/25/2019
Research online shows that the largest engaged audiences for The Guide to the Constitution offer for The Heritage Foundation skews towards a 50+ age. With that in mind, we wanted to see if enlarging the text would increase email acquisition rate for the offer.
Research Question
Would larger text increase email acquisition rate?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 2.7% | ||
| T1: | Treatment #1 | 2.2% | -17.8% | 98.0% |
This experiment has a required sample size of 7,862 in order to be valid. Since the experiment had a total sample size of 22,298, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 17.8% decrease in conversion rate
× 0% increase in average gift
Key Learnings
With a 98% level of confidence, we identified that the larger text decreased email acquisition rates for the offer by -17.8%.
Our hypothesis is that although the content may have been easier to read, the positive of improved readability was overtaken by the increased friction of increased scrolling required to signup for the offer.
Further experimentation may be required to further validate, but it is advisable to not assume that increased font sizes to improve readability of a page will improve overall performance for the page.