How adding a donate button to the header and navigation menu affects donor conversion rate
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 11/15/2023 - 12/11/2023
The Daily Signal is The Heritage Foundation’s multimedia news platform. On their website, the only call-to-action to donate and support the Daily Signal’s work is located at the very bottom in the footer of each page and doesn’t stand out.
In an attempt to increase the amount of traffic to the donation page from the home page, we wanted to test if adding a more prominently displayed donate button to the top of the home page led to an increase in donor conversion rate.
The donate button appeared in both the home page’s header and dropdown navigation menu (in both the desktop and mobile experiences) and was blue in color in order to stand out from the surrounding buttons. 50% of home page traffic saw the control and 50% saw the treatment.
Research Question
We believe that adding a donate button to the header and navigation menu for home page visitors will achieve an increase in donor conversion rate because more people will click to view the donation page. because more people will click to view the donation page.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | 0.02% | |||
| T1: | Donate button in header | 0.05% | 99.9% | 80.3% |
This experiment has a required sample size of 49,328 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
This experiment ran for 27 days, in which time the donate button variant saw a 42.8% increase in donations for all traffic compared to the control. However, it is important to note that the statistical level of confidence for this increase is 53.2%, meaning that the results cannot be considered valid or reliable for future decision making.
In terms of revenue, the treatment saw a decrease of 0.11% in average gift, but at a 6.52% level of confidence these results cannot be relied upon to predict a valid outcome.
Reasons for this may include the donate button appearing less hierarchically prominent than the email “Sign Up” button above it, or home page visitors not yet having experienced the value in the Daily Signal’s articles before seeing a donate button.
Based on the results, it is clear that additional testing is needed to understand what motivates The Daily Signal visitors to donate to the organization. This could include adding value proposition copy next to the button, or testing different locations on the home page. It may also be worth testing an unobtrusive sticky donate button that follows the visitor through their journey sitewide.