How did the use of an exit popup increase donor conversion rates from The Daily Signal visitors?
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Ended On: 01/02/2019
On The Daily Signal’s year-end fundraising page in 2018, we installed an exit popup with two messages, where one was focused on “Stand with the President” while the other was “Stand with Heritage”.
Research Question
We had two questions:
- How did the use of an exit popup increase donor conversion rates?
- Which message would convert the most donors (stand with either (a) the president, or (b) Heritage)?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | "Stand with the president" Message | 5.4% | ||
| T1: | "Stand with Heritage" Message | 5.9% | 8.4% | 21.1% |
This experiment has a required sample size of 19,949 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
Although the experiment did not validate, we gleaned some insight from the experiment.
Exit Popup Tactic Observations:
It’s also worth noting that the landing page secured a total of 27 transactions without firing the “exit popup”, and that the “exit popup” regardless of messaging variant secured an additional 42 gifts combined between the two message types. That means that the exit popup seemingly secured an additional 55% more gifts from donation page viewers who were attempting to leave the page. Subsequently, the revenue from the “exit popup audience” drove an incremental 58% in additional revenue to the audience that did not see the popup.
This supports the idea that the exit popup does indeed work to secure additional transactions and revenue.
Messaging Observations:
- Although the “Stand with Trump” message actually had a 14% higher click rate on the affirmative button (40 / 350 = 11.4% CTR) compared to the “Stand with Heritage” message (which was 38 / 391 = 9.7% CTR).
- However, we actually saw that the “Stand with Heritage” message drove an increase of 8.4% for donor conversions (5.9% vs. 5.4%).
- Finally, The “Stand with Heritage” also had a 12.6% higher average gift value ($45.81 vs. $40.67) compared to the “Stand with Trump” message.
Final Conclusion:
With only 21% of the required sample size collected during this experiment, we propose continuing to test which messages work best to drive donor conversions, but that it would certainly appear that installing an exit popup on donation pages can have a profound impact as a tactic to convert donors and drive incremental revenue that otherwise would leave the donation page without interruption.