How email copy length affected click-through-rate on an appeal
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Ended On: 04/05/2021
For a membership renewal campaign, we wanted to look at ways to optimize the email copy for The Heritage Foundation appeals. The test is intended to determine whether shorter or lengthier email copy results in higher click-through-rates. The version with the higher click-through-rate sends more users to the landing page linked in the email.
The control email consists of seven short paragraphs while the treatment email consists of five short paragraphs. The sender and subject line are identical for each email, and the “Renew” and “Sign up to renew” links are included in each email.
Research Question
We believe that shortening copy for housefile emails will achieve improved click-through-rates.
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 4.8% | ||
| T1: | Treatment #1 | 5.2% | 7.2% | 80.1% |
This experiment has a required sample size of 29,775 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
Although the experiment resulted in a 7.2% increase in clicks, the level of confidence was only 80%. The treatment (shorter copy) did ultimately generate more clicks than the control (lengthier copy). To further increase the level of confidence, it may be wise to conduct another test in which the shorter copy is paired with a personalized salutation.