Reasons to Believe

How a forward style email in a high urgency campaign affects giving

Experiment ID: #123472

Reasons to Believe

Experiment Summary

Ended On: 01/17/2023

In an effort to continue to humanize communications during a high urgency campaign, we wanted to use a “forward” style email in one of our email appeals. This email would refer to a previous email and make it seem as if the email was forwarded to the recipient. The control for this experiment had the intended, original email (a message from the President). The treatment was also a message from the President but it asked the email recipient to refer to the previous email sent.

In both of these emails previous donors were suppressed so as not to accidentally include those who may have given in the previous email. The email tested was an evening email on the last day of the campaign.

Research Question

We believe that using a forward style email for prospective donors will achieve a higher donor conversion rate .

Design

C: Control
T1: Treatment #1

Results

  Treatment Name Conv. Rate Relative Difference Confidence
C: Control 0.16%
T1: Treatment #1 0.16% 3.2% 6.7%

This experiment has a required sample size of 4,885,060 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

This experiment resulted in a 3.6% increase in donor conversion rate with a level of confidence of 6.69%. This means that this experiment had almost no effect on donor conversion rate. The control seemingly has a higher average gift that is skewed by a $3k gift. There is also no noticeable change in click thru rate. Our hypothesis is that because this was tested in an email that’s in the “high urgency” phase of the campaign, motivation to give before the deadline is already high. It’s recommended that this is retested at a different phase in a campaign to more clearly discern motivation.


Experiment Documented by NextAfter

Question about experiment #123472

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.