How adding donor cultivation stories on top of a campaign appeal schedule of emails impacted revenue
Leadership Institute
Experiment Summary
Timeframe: 12/10/2020 - 01/02/2021
The Leadership Institute decided to experiment with adding donor stories on top of their regularly scheduled calendar year-end campaign email appeal schedule to experiment with whether or not it inspired more people to give at all or increased generosity from those that saw the donor story emails.
To administer the test, we evenly split the file based upon (a) donor vs. non-donor statuses, (b) the donor’s highest previous contribution tier, (c) how long the person was on the email file, and finally (d) whether or not they were currently considered “engaged” or “disengaged” from the email program (where “engaged” was whether or not they had opened any of the last 10 emails that we had sent to them before splitting the email file evenly for this experiment).
The donor stories were sent from a “secondary sender” for the organization—someone who was planned to be used in two of the year-end campaign messages.
Finally, to ensure that the samples within the study are accurate, we looked at unique individual emails for both segments that received any year-end email (including the “donor story” emails for the treatment segment)
Research Question
We believe that sending donor stories for year-end campaign recipients will achieve increase revenue.
Design
Results
| Treatment Name | Revenue per Visitor | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | $2.40 | $161.66 | ||
| T1: | Treatment #1 | $2.35 | -1.9% | 53.8% | $206.55 |
This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was not met so these experiment results are not valid.
Key Learnings
With only a 53.7% level of confidence, we observed that the treatment segment did not increase revenue in any significant way.
Although this treatment segment saw a decrease in the donor conversion rate of -23.2% (with a valid level of confidence of 99.4%) however, we don’t believe that the additional donor story emails drove the decrease in donor conversion rate.
We did notice, however, that there was a +27.8% average gift value for those donors that did decide to give from the treatment segment over that of those in the control segment (who did not receive the donor stories).
Further experimentation should be considered with this approach.