How adding urgency copy w/ a specific ask affects donations
e3 Partners
Experiment Summary
Timeframe: 03/30/2022 - 04/04/2022
I Am Second wanted to see if changing the copy to have a sense of urgency with a specific ask would increase donations when added to a pop up. For the high urgency fiscal year end (FYE) campaign, IAS ran an experiment with 2 versions of a sticky bar pop up that asked for support.
In the control version, the copy asked if the potential donor would make a donation by Midnight on March 31st (IAS FYE). In the treatment version, there was specific copy that asked for “10 people to give $50 by midnight”.
The goal was to see if using a smaller believable amount of people and $ amount needed by a specific time would lead to more donations.
Research Question
We believe that asking for a specific $ amount and specific number of donors for People visiting the website will achieve A higher number of donations than a less specific ask..
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | FYE22: IAS SB - Control | 0.48% | $0.00 | ||
| T1: | FYE22: IAS SB - Variant (Urgency) | 1.00% | 106.7% | 77.4% | $0.00 |
This experiment has a required sample size of 2,116 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
While the experiment did not reach statistical validity due to not enough sample size, we did have some positive learnings. We saw a 106.7% increase in donations overall during this very short timeframe. While we saw a 77% Level of Confidence in these results, we did not have a large enough sample size to draw any concrete resolutions. However, we saw enough directional lift to say that this tactic would be worth experimenting with again on future campaigns.
Changes to future experiments may include a phased approach to urgency, where we start with a more generic ask similar to the control, then switch to a version similar to the treatment version with more specificity closer to the close of a high urgency campaign…to have more credibility. After-all…urgency cannot be sped up, only highlighted.
Notable changes to other metrics due to the experiment:
- Revenue experienced a 275.3% increase with a 96% level of confidence.
Observations of the impact on other visitor segments in the experiment:
- Organic Search visitors had a 2.0% increase in donations with a 92% level of confidence.