How a cultivation email prior to an appeal impacted giving Experiment ID: #15371
Buckner International is a global ministry dedicated to the transformation and restoration of the lives we serve. We are a Christ-centered organization that delivers redemptive ministry to the most vulnerable from the beginning to the ending of life.
Timeframe: 4/25/2019 - 5/5/2019
In an analysis of prior appeals, Buckner International had discovered that their long story emails tended to be the lowest performing appeals. However, we had a hypothesis that while these emails may not perform well immediately, they may be influencing downstream giving. Given the time and effort required to write these emails, we decided to run a test to measure the impact of these “cultivation” style appeals when they are a part of a larger campaign.
To do this, we split the file in half and sent one segment a long story prior to the two appeal emails. The treatment just received the two appeal emails. We then monitored the donor giving across the entire campaign.
Will long-form story emails positively impact overall giving to a campaign?
|Treatment Name||Conv. Rate||Relative Difference||Confidence|
|C:||Long Story Pre-Email||0.03%|
|T1:||No Story Email||0.03%||-12.6%||26.2%|
This experiment has a required sample size of 1,378,748 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
At the end of the campaign, we did not see any significant difference in donor conversion with the additional email. However, we did see a 50% increase to unsubscribes (with 100% level of statistical confidence) due to the addition of the extra email. Given that the additional story email did not have the positive impact on giving that we hypothesized, it would seem that we can reduce the number of emails in the appeal while negatively impacting giving.