How additional cultivation impacts online giving Experiment ID: #18583

Alliance Defending Freedom

Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.

Experiment Summary

Timeframe: 4/12/2019 - 10/12/2019

During a data analysis performed for ADF at the beginning of the calendar year, we discovered a handful of trends related to the email file that were concerning:

  • A significant portion ADF subscribers had not opened an email within the last six months
  • New and existing donors were more likely to disengage as non-donor subscribers
  • ADF was sending twice as many solicitation emails as it was cultivation and stewardship-focused emails. Simply put: the subscribers were being asked twice as much as they were given back to.

While the digital fundraising program had been growing, these trends illustrated a potential problem in the near future if they were not corrected. The historical success of ADF’s online fundraising was closely correlated to the size and engagement of the email file.  

Given the ambitious fundraising goals, we couldn’t reduce the number of solicitations. So, instead, we decided to test out an additional cultivation series that would increase the number of emails sent to the file. The additional emails would focus on educating and feeding back into the various segments. There would be no additional solicitations.

At a tactical level, this email series consisted of simply-designed emails sent each Friday by one of the authors on the blog (Sarah K). They would include a link to a blog post or a video and we emphasized getting replies to the email.

To validate the impact of this series, we split the email file in half, paying special attention to ensure that there was equal representation of active, lapsed and non-donors and that the average revenue donor was similar between the splits.

Research Question

Will an increase in cultivation emails have any impact on a donor’s likelihood to give?

Design

C: No Additional Cultivation
T1: Additional Cultivation Group

Results

Treatment Name Revenue per Visitor Relative Difference Confidence Average Gift
C: No Additional Cultivation $9.61 $179.77
T1: Additional Cultivation Group $13.60 41.5% 99.9% $225.00

This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was met so these experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 13.1% increase in conversion rate
× 25.2% increase in average gift
41.5% increase in revenue

Key Learnings

After running for a full six months, we observed a statistically significant 13.1% increase in the likelihood for a donor to give a gift online. Combined with the 25% increase in average gift, this resulted in a 41.5% increase in online revenue from the donors.

While the emphasis of this experiment was to measure the impact of online giving and engagement, we also observed several other trends:

  • The impact of the cultivation did not fade over time. We observed the same statistically significant increase in likelihood to give and revenue when we compared the first three months of giving to the second three months.
  • The additional cultivation did not significantly impact a donor’s likelihood to give across all channels but we did see an increase in the amount they gave by 30% regardless of channel (with a 99% level of confidence.) It should be noted that this lift did not reach the required sample size so this should be considered a directional lift.
  • The additional cultivation did not increase the likelihood of a non-donor to make their first gift. However, it did increase their average revenue per donor by 50% in the 6 month period.

* For all of the statistics above, we excluded both gifts above $10K and all recurring gifts.


Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.