How a cultivation email prior to an appeal impacted giving Experiment ID: #15371

Buckner International

Buckner International is a global ministry dedicated to the transformation and restoration of the lives we serve. We are a Christ-centered organization that delivers redemptive ministry to the most vulnerable from the beginning to the ending of life.

Experiment Summary

Timeframe: 4/25/2019 - 5/5/2019

In an analysis of prior appeals, Buckner International had discovered that their long story emails tended to be the lowest performing appeals. However, we had a hypothesis that while these emails may not perform well immediately, they may be influencing downstream giving. Given the time and effort required to write these emails, we decided to run a test to measure the impact of these “cultivation” style appeals when they are a part of a larger campaign.

To do this, we split the file in half and sent one segment a long story prior to the two appeal emails. The treatment just received the two appeal emails. We then monitored the donor giving across the entire campaign.

Research Question

Will long-form story emails positively impact overall giving to a campaign?


C: Long Story Pre-Email
T1: No Story Email


Treatment Name Conv. Rate Relative Difference Confidence
C: Long Story Pre-Email 0.03%
T1: No Story Email 0.03% -12.6% 26.2%

This experiment has a required sample size of 1,378,748 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

At the end of the campaign, we did not see any significant difference in donor conversion with the additional email. However, we did see a 50% increase to unsubscribes (with 100% level of statistical confidence) due to the addition of the extra email. Given that the additional story email did not have the positive impact on giving that we hypothesized, it would seem that we can reduce the number of emails in the appeal while negatively impacting giving.

Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.