How using a homepage takeover vs a pop up impacted giving during a high urgency campaign
Reasons to Believe
Experiment Summary
Timeframe: 06/28/2022 - 07/04/2022
Reasons to Believe was running a high urgency campaign (Fiscal Year End) and wanted to drive more donations during the last week of the campaign. In previous campaigns RTB has done a homepage takeover or a pop up at the end of the campaign to direct website traffic to the donation page. In order to discern which method works best for their audience we decided to run a test where half of their site visitors would get a homepage takeover and the other half would get a pop up. Our initial hypothesis is that more web visitors would convert to donors on a homepage takeover because it takes up more real estate and in turn draws more attention to the campaign. A pop up can be easily dismissed on a webpage.
Research Question
We believe that using a home page takeover during a high urgency campaigh for website visitors will achieve a higher donor conversion rate .
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Homepage Takeover | 1.6% | $0.00 | ||
| T1: | Homepage Popup | 3.0% | 82.3% | 77.7% | $0.00 |
This experiment has a required sample size of 965 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
When looking at all traffic, this experiment resulted in a 21% lift in donor conversion rate but with only a level of confidence of 40%. When we observe only new visitors we see a higher level of confidence: 82% lift with a level of confidence of 77%. While this did not validate, it is probable that new visitors to the site may have been confused by seeing an entirely new landing page (the Fiscal Year End donation page) and therefore responded more to the pop up. However, since this experiment only ran for the last week of the campaign, we did not see sufficient enough traffic to validate the results. It is recommended that this is retested over a longer period of time.