The impact of a membership offer on organic website visitors Experiment ID: #16781

Alliance Defending Freedom

Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.

Experiment Summary

Timeframe: 5/16/2019 - 6/26/2019

In the continued efforts to measure the potential impact of membership on the ADF audience, we devised an experiment to be run on the homepage of the main website. For the treatment version, we added a small ad below the main website banner and an additional navigational button to “Become a Member.” For the control version, we left the navigation the same and added an ad below the main website banner for an acquisition page that was comparable* to the membership page. 

To validate this experiment, we targeted all visitors landing on the home page that did not have a recent gift. To be consistent, once an individual was in the treatment group, they would continue to see the “Become a Member” navigational option as they moved through the site.

* The control acquisition page was comparable in that it consisted of several stories about ADF and its clients, had similar imagery, and culminated in an ask for a donation. The page used was the “Stories of Faithfulness” offer.

Research Question

Will a “membership” offer acquire more gifts than an offer that asks for a generic donation?


C: Homepage Banner - Donate Treatment
T1: Homepage Banner - Membership Treatment


Treatment Name Conv. Rate Relative Difference Confidence Average Gift
C: Homepage Banner - Donate Treatment 0.71% $0.00
T1: Homepage Banner - Membership Treatment 0.93% 30.9% 95.1% $0.00

This experiment has a required sample size of 12,916 in order to be valid. Since the experiment had a total sample size of 26,450, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 30.9% increase in conversion rate
× 0% increase in average gift

Key Learnings

After running for a month and a half, we found that the membership offer resulted in a 31% increase to donations. For this experiment, we included only donations that were made to either the general donation form or one of the offer donation pages (Stories of Faithfulness vs. Membership.) When will drill down, we found that the general donations were not statistically impacted; the big lift came from the gifts given to “membership” vs. the “stories of faithfulness” offer.

A couple other notes:

  • It is important to remember that this experiment purposefully excluded known donors. This means the 31% lift was primarily with new donors.
  • The new visitors were more likely to respond to the membership offer than returning visitors. In fact, new visitors were 375% more likely to donate to the membership offer.
  • The experiment did not have any statistical impact on the likelihood of the visitor to make a general donation via the main donation button.

Given the length of time needed to validate this experiment on the homepage, the next recommended step is to move toward paid media (Facebook) to accelerate our learnings.

Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.