How replacing older donation page copy impacted donor conversion rate for a new offer
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Timeframe: 11/01/2019 - 12/13/2019
When launching a new content offer related to the impeachment of the president and how the process works, we decided to launch with to donation page treatments–one that had been used as the donation page for a short series of emails when the news was released, and another that had not yet been tested. This approach was used to mitigate risk with new donation page related to the topic, and since the housefile emails along with the control donation page had produced some nice results, we assumed we would be able to capitalize upon this page’s positioning of the fundraising appeal for new content downloaders.
Research Question
Would reusing the donation page copy previously used to send a series of appeals work behind a new content offer for a related subject?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 11.2% | ||
| T1: | w/Conversation | 14.1% | 25.1% | 85.5% |
This experiment has a required sample size of 1,063 in order to be valid. Since the experiment had a total sample size of 1,181, and the level of confidence is not above 95% the experiment results are not valid.
Key Learnings
The treatment page (which was previously unseen by the housefile) showed an aggregate increase in donor conversion rate of 25.1%, but did not validate (LoC: 85.5%) because we turned it off early.
Why? Because we actually saw no difference between the treatments for new visitors, but saw a decline in donor conversion rate of 34% (LoC: 92%) for returning visitors to the control page.
Our hypothesis is that donors that had previously seen the control page through the earlier series of appeals felt like they had already given a gift in response to that ask. For this reason, they decided not to give another gift to the page, but for new visitors–the treatment performed slightly better (whereas the control message had no difference).
For that reason, we decided to roll out the treatment copy as the new control, as it showed no negatives for any audience type, which will remove the risks associated with returning visitors feeling like they have already given to “this initiative,” and presenting them with a new ask instead.