How the sender did not impact donor conversion rates for prospecting emails
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 01/28/2019 - 01/31/2019
With the senders in the 2018 Kickoff Campaign messaging no longer with the organization, we wanted to identify who should be the primary sender for consideration on the 2019 Kickoff Campaign messages through the email prospecting channel.
Research Question
Does the sender of the Kickoff Campaign messaging impact donor conversion rates at all?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Sender: Andrew | 0.03% | ||
| T1: | Sender: Maggy | 0.03% | -1.3% | 7.5% |
This experiment has a required sample size of 143,116,923 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
With regards to the Kickoff Campaign in 2019, we found that through the email prospecting channel, that the sender from Heritage played no role in donor conversion rates. As shown, we see a -1.3% difference between the two senders, but the level of confidence is only 7.52% — nowhere near the 95% validation that we’re looking for.
This was the case with the other performance metrics we would subsequently be interested in, as well, which were effected as follows:
- Email Clickthrough Rate: +2.9% for Maggy (over Andrew), with a 41.9% LoC.
- Email Open Rate: +0.5% for Maggy (over Andrew), with a 48.9% LoC.
Any additional experimentation was turned off, and Andrew was used for the remainder of the campaign sends within the timeline.
The key takeaway from this experiment is that the Heritage Foundation’s file is so accustomed to these cyclical campaigns, and that messaging for the campaign is very transactional in nature, which focuses upon the primary motivation of: “Renewing” or “Activating” your membership for 2019, that the sender, at least in the email prospecting channel, is not important.