The impact of a membership offer on organic website visitors
Alliance Defending Freedom
Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.
Experiment Summary
Timeframe: 05/16/2019 - 06/26/2019
In the continued efforts to measure the potential impact of membership on the ADF audience, we devised an experiment to be run on the homepage of the main website. For the treatment version, we added a small ad below the main website banner and an additional navigational button to “Become a Member.” For the control version, we left the navigation the same and added an ad below the main website banner for an acquisition page that was comparable* to the membership page.
To validate this experiment, we targeted all visitors landing on the home page that did not have a recent gift. To be consistent, once an individual was in the treatment group, they would continue to see the “Become a Member” navigational option as they moved through the site.
* The control acquisition page was comparable in that it consisted of several stories about ADF and its clients, had similar imagery, and culminated in an ask for a donation. The page used was the “Stories of Faithfulness” offer.
Research Question
Will a “membership” offer acquire more gifts than an offer that asks for a generic donation?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Homepage Banner - Donate Treatment | 0.71% | ||
| T1: | Homepage Banner - Membership Treatment | 0.93% | 30.9% | 95.1% |
This experiment has a required sample size of 12,916 in order to be valid. Since the experiment had a total sample size of 26,450, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 30.9% increase in conversion rate
× 0% increase in average gift
Key Learnings
After running for a month and a half, we found that the membership offer resulted in a 31% increase to donations. For this experiment, we included only donations that were made to either the general donation form or one of the offer donation pages (Stories of Faithfulness vs. Membership.) When will drill down, we found that the general donations were not statistically impacted; the big lift came from the gifts given to “membership” vs. the “stories of faithfulness” offer.
A couple other notes:
- It is important to remember that this experiment purposefully excluded known donors. This means the 31% lift was primarily with new donors.
- The new visitors were more likely to respond to the membership offer than returning visitors. In fact, new visitors were 375% more likely to donate to the membership offer.
- The experiment did not have any statistical impact on the likelihood of the visitor to make a general donation via the main donation button.
Given the length of time needed to validate this experiment on the homepage, the next recommended step is to move toward paid media (Facebook) to accelerate our learnings.