Does a high urgency campaign with a direct ask and goal drive more results than general Christmas gift messaging?
CRISTA Ministries
Experiment Summary
Timeframe: 12/01/2018 - 12/20/2018
World Concern’s Global Gift Guide email strategy has traditionally focused upon messaging donors leading up to Christmas in a very transactional way.
As a part of the 2018 strategy, to test into creating a “High Urgency Campaign” that pulled out 4 of the most popular gifts (with the highest appeal and organizational mission alignment) and created a fundraising goal, with challenge messaging to challenge the donor to help us deliver specific gifts by a deadline.
Research Question
Does a high urgency campaign with a direct ask and goal drive more results than general Christmas gift messaging?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 9.4% | ||
| T1: | Treatment #1 | 18.8% | 99.4% | 99.8% |
This experiment has a required sample size of 97 in order to be valid. Since the experiment had a total sample size of 588, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 99.4% increase in conversion rate
× 0% increase in average gift
Key Learnings
The high urgency campaign messaging treatment essentially doubled the donor conversion rate (+99.4% increase) when compared to the transactional gift messages that had been traditionally used leading up to Christmas giving deadlines.
Typically, revenue lift experiments require far more than 75 transactions to validate, but we did also achieve a statistical significance in increasing revenue for the campaign through donations received through the treatment vs. the control. With a 96.5% level of confidence, we also saw an increase in the revenue per visitor statistic by 71.3%.
We would recommend further testing (and gathering far more than 75 transactions) to have full confidence with the revenue lift aspect of this type of experiment, but when combining both revenue lift potential with twice as many gifts, leveraging high urgency campaigns over transactional messages within the confines of a holiday gift catalog program appears to drive a dramatic improvement.