How adding urgency copy w/ a specific ask affects donations - NextAfter
e3 Partners

How adding urgency copy w/ a specific ask affects donations

Experiment ID: #89668

e3 Partners

Experiment Summary

Timeframe: 03/30/2022 - 04/04/2022

I Am Second wanted to see if changing the copy to have a sense of urgency with a specific ask would increase donations when added to a pop-up. For the high urgency fiscal year-end (FYE) campaign, IAS ran an experiment with 2 versions of a sticky bar pop-up that asked for support.

In the control version, the copy asked if the potential donor would make a donation by Midnight on March 31st (IAS FYE). In the treatment version, there was a specific copy that asked for “10 people to give $50 by midnight”.

The goal was to see if using a smaller believable amount of people and dollar amount needed by a specific time would lead to more donations.

Research Question

We believe that asking for a specific dollar amount and a specific number of donors for people visiting the website will achieve a higher number of donations than a less specific ask.

Design

C: FYE22: IAS SB - Control
T1: FYE22: IAS SB - Variant (Urgency)

Results

  Treatment Name Conv. Rate Relative Difference Confidence Average Gift
C: FYE22: IAS SB - Control 0.48% $0.00
T1: FYE22: IAS SB - Variant (Urgency) 1.00% 106.7% 77.4% $0.00

This experiment has a required sample size of 2,116 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

While the experiment did not reach statistical validity due to not enough sample size, we did have some positive learnings. We saw a 106.7% increase in donations overall during this very short timeframe. While we saw a 77% Level of Confidence in these results, we did not have a large enough sample size to draw any concrete resolutions. However, we saw enough directional lift to say that this tactic would be worth experimenting with again on future campaigns.

Changes to future experiments may include a phased approach to urgency, where we start with a more generic ask similar to the control, then switch to a version similar to the treatment version with more specificity closer to the close of a high urgency campaign…to have more credibility. After all…urgency cannot be sped up, only highlighted.

Notable changes to other metrics due to the experiment:

  • Revenue experienced a 275.3% increase with a 96% level of confidence.

Observations of the impact on other visitor segments in the experiment:

  • Organic Search visitors had a 2.0% increase in donations with a 92% level of confidence.


Experiment Documented by NextAfter

Question about experiment #89668

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.