How removing an ad’s premium offer affects donations
Texas State Historical Association
Experiment Summary
Timeframe: 12/27/2015 - 01/04/2016
The Texas State Historical Association, like many other nonprofits, utilized a banner across their site during their year end campaign to promote an opportunity to give. Historically, all of the banners on TSHA’s sites had been utilized to promote premiums such as eBooks so it made logical sense to promote the premium associated with becoming a member (the Texas Almanac).
However, we realized this premium may not appeal to everyone so we decided to create a treatment that would test out a purely philanthropic ask on the banners.
Research Question
What banner will motivate website visitors to become a member during a calendar year end campaign?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Premium Offer | 0.04% | $54.04 | ||
| T1: | Philanthropic Offer | 0.02% | -52.6% | 94.1% | $114.08 |
This experiment has a required sample size of 57,533 in order to be valid. Since the experiment had a total sample size of 107,006, and the level of confidence is not above 95% the experiment results are not valid.
Key Learnings
We discovered that the Almanac premium proved to be a significant motivator for donors. There was no statistically significant difference in the amount of traffic driven by the ad but those that were driven by the premium offer were 52.6% more likely to become a member.
Another interesting aspect of this experiment revolves around the average gift amount of the two audiences. While not statistically significant, the average gift of the philanthropically driven donors was over double that of the premium driven donors. It is worth revisiting this experiment in future campaigns to see if this trend holds.