How a different offer affects email acquisition and cost Experiment ID: #21721


Experiment Summary

Timeframe: 5/20/2020 - 6/3/2020

In this experiment, we had an idea to promote 2 different offers related to donation pages. The first – an in-depth research study on donation pages – is the “higher quality” offer is there’s more research, data, examples, and tactics. But we wondered if our general donation page template might acquire more emails since it is simple, actionable, and requires less work to digest.

Research Question

Will a different type of offer lead to better email conversion rates?


Treatment Name Conv. Rate Relative Difference Confidence
C: State of Nonprofit Donation Page Study 0.68%
T1: General Donation Page Template 1.2% 70.2% 100.0%

This experiment has a required sample size of 3,463 in order to be valid. Since the experiment had a total sample size of 85,280, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 70.2% increase in conversion rate
× 0% increase in average gift

Key Learnings

The general donation page template was the overwhelming winner, despite an even sample size because of Facebook’s ad prioritization. It not only led to a 70% increase in conversion rate, but it was also 50% cheaper to acquire an email address.

A brief analysis of the quality of contacts indicated the general donation page emails may even be higher quality than the leads coming from the study.

Experiment Documented by...

Nathan Hill

Nathan is an Optimization Evangelist at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.