How addressing a new segment of donors on the homepage impacted donor conversion
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 03/21/2017 - 04/07/2017
When The Heritage Foundation redesigned their website, we ran several experiments on the new homepage in an effort to optimize for donor conversion. Our initial test was designed to make the donation and membership renewal buttons more prominent on the page. We also included some value proposition copy along with the buttons to help sell both a donation and membership renewal. However, after running for a month, this test did not produce a significant impact on donor conversion.
After further analysis, we developed two new hypotheses. We realized that we were asking the visitors to make a choice between two equally weighted options. When doing this, it will often lead to decision paralysis meaning they end up not selecting either option. Additionally, we realized that there was a third potential donor segment that we were not addressing with the homepage. For The Heritage Foundation, all donors are considered a member. As a result, most of the messaging is around becoming a member yet none of the calls to action were addressing the potential donor audience.
We decided to create a new treatment combining both hypotheses using the same callout box tactic but focusing the messaging and call to action around becoming a member.
Research Question
Will creating messaging and a call to action about becoming a member increase the overall donor conversion on the page?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | No Callout | 0.20% | $82.49 | ||
| T1: | Join Heritage Callout | 0.29% | 46.4% | 99.0% | $98.85 |
This experiment has a required sample size of 21,867 in order to be valid. Since the experiment had a total sample size of 74,868, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 46.4% increase in conversion rate
× 19.8% increase in average gift
75.4% increase in revenue
Key Learnings
By addressing the segment of visitors that were interested in making their first gift to become a member, we were able to boost the overall donor conversion for people coming to the homepage by 46%. When analyzing these results further, we realized that the call out message resulted in a 74% increase in traffic to the various donation pages. We were driving more non-donors to the page which decreased their overall motivation but it resulted in a net increase to the number of visitors that eventually made a gift.