How asking for less impacted year end giving
KCBI
Experiment Summary
Timeframe: 12/01/2021 - 12/07/2021
During the beginning of their year end campaign, KCBI decided to see if they could statistically validate a concept they had stumbled upon during a previous high urgency campaign. The idea was that if we asked existing donors for less than their previous highest gift amount and gave them a reason for doing so, we could inspire more people to give overall.
Originally, our belief was that it may lower average gift but would increase response rate enough to improve overall revenue. However, we found that it was able to increase both average gift and response rate for the segment but did not reach the 95% level of statistical validity we look for when determining a winner. With this in mind, we tested the same concept again at the start of their year end campaign.
For this test, we divided the file between control and treatment. For the control, we used email personalization to ask them for a specific amount (based upon a previous highest gift amount) and identified that we’re looking for a specific number of donors. For example, if a donor had given $20 in the past, we would ask: “Would you consider being one of 50 donors to make a gift of $25 right now.”
For the treatment, we used the same logic but asked them for a level down compared to the control. For example, if a donor had given $20, we would ask them to be one of 50 donors to make a $10 gift right now.”
Research Question
We believe that asking for less than their previous highest amount during a year end fundraising campaign will increase overall revenue and response.
Design
Results
| Treatment Name | Revenue per Visitor | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | $0.65 | $54.09 | ||
| T1: | Treatment #1 | $1.91 | 193.9% | 99.8% | $106.77 |
This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was met so these experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 48.9% increase in conversion rate
× 97.4% increase in average gift
193.9% increase in revenue
Key Learnings
After sending three separate emails to the distinct segments using similar language in each language and maintaining the same control / treatment split through the entire series, we discovered that the treatment group saw a 193.9% increase to overall revenue. This came from increasing conversion rate by 49% and average gift by 97%.
When digging into the results, we found that the biggest revenue increase came from donors giving between $250 and $500. The treatment group had eight donors giving in that range while the control had none.
It should also be noted that we removed a $2,600 gift from the results since it was such a large outlier when compared to the other gifts. When it is included, we observed a 39% increase with the treatment with a 73% level of confidence.