How a different offer affects email acquisition and cost
NextAfter
Experiment Summary
Timeframe: 05/20/2020 - 06/03/2020
In this experiment, we had an idea to promote 2 different offers related to donation pages. The first – an in-depth research study on donation pages – is the “higher quality” offer is there’s more research, data, examples, and tactics. But we wondered if our general donation page template might acquire more emails since it is simple, actionable, and requires less work to digest.
Research Question
Will a different type of offer lead to better email conversion rates?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | State of Nonprofit Donation Page Study | 0.68% | ||
| T1: | General Donation Page Template | 1.2% | 70.2% | 100.0% |
This experiment has a required sample size of 3,463 in order to be valid. Since the experiment had a total sample size of 85,280, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 70.2% increase in conversion rate
× 0% increase in average gift
Key Learnings
The general donation page template was the overwhelming winner, despite an even sample size because of Facebook’s ad prioritization. It not only led to a 70% increase in conversion rate, but it was also 50% cheaper to acquire an email address.
A brief analysis of the quality of contacts indicated the general donation page emails may even be higher quality than the leads coming from the study.