How a forward style email in a high urgency campaign affects giving
Reasons to Believe
Experiment Summary
Ended On: 01/17/2023
In an effort to continue to humanize communications during a high urgency campaign, we wanted to use a “forward” style email in one of our email appeals. This email would refer to a previous email and make it seem as if the email was forwarded to the recipient. The control for this experiment had the intended, original email (a message from the President). The treatment was also a message from the President but it asked the email recipient to refer to the previous email sent.
In both of these emails previous donors were suppressed so as not to accidentally include those who may have given in the previous email. The email tested was an evening email on the last day of the campaign.
Research Question
We believe that using a forward style email for prospective donors will achieve a higher donor conversion rate .
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.16% | ||
| T1: | Treatment #1 | 0.16% | 3.2% | 6.7% |
This experiment has a required sample size of 4,885,060 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
This experiment resulted in a 3.6% increase in donor conversion rate with a level of confidence of 6.69%. This means that this experiment had almost no effect on donor conversion rate. The control seemingly has a higher average gift that is skewed by a $3k gift. There is also no noticeable change in click thru rate. Our hypothesis is that because this was tested in an email that’s in the “high urgency” phase of the campaign, motivation to give before the deadline is already high. It’s recommended that this is retested at a different phase in a campaign to more clearly discern motivation.