How changing the sender name affected the open rate on an email appeal
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Ended On: 04/05/2021
In an effort to increase overall email engagement, we wanted to test different email senders to measure open rate. We wondered if users would be more inclined to open an email from one sender than another. The control sender was “Maggy Smith” and the treatment sender was “Andrew P. McIndoe.” Maggy works in Membership Programs and Andrew is the Vice President of Development. Each email contains the same subject line and body copy.
Research Question
We believe that changing the sender for housefile email recipients will achieve increased open rates.
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 21.5% | ||
| T1: | Treatment #1 | 21.4% | -0.6% | 39.0% |
This experiment has a required sample size of 815,329 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
The experiment resulted in a -0.59 decrease in traffic and a 39.0% level of confidence. Users engaged less with Andrew P. McIndoe than they did with Maggy Smith. A possible reason is users are more familiar with Maggy Smith, as they’ve heard from her before about membership relations. Also, an email from a representative of Membership Programs may be more engaging than an email from the Vice President of Development, assuming the user is familiar with each.
The next experiment could be to change the subject line to something more specific, such as “Renew your membership today” rather than “I haven’t heard from you.” The user may be more inclined to open the email because of a needed action, regardless of the sender.