How promoting an offer in the Dear Reader ad copy instead of membership increased donor conversion rate Experiment ID: #18185

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Timeframe: 9/9/2019 - 9/14/2019

In previous experimentation with the “Dear Reader” ads for The Daily Signal, we had noticed that there was a consistent finding of 5x to 10x more click activity on the ad when we would change the call to action from “Activate your membership” to “Learn more.” This told us that there was a lot of additional click activity available to us that we were not capturing.

In an attempt to see if this would translate into increased giving and revenue, we experimented with promoting a new offer instead of membership in this ad for The Heritage Foundation.

Research Question

Does promoting an offer instead of membership increase giving or revenue?

Design

C: THF Membership Ask (Control)
T1: Offer Promotion (Treatment)

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: THF Membership Ask (Control) 0.05%
T1: Offer Promotion (Treatment) 0.41% 803.8% 100.0%

This experiment has a required sample size of 1,356 in order to be valid. Since the experiment had a total sample size of 18,442, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 803.8% increase in conversion rate
× 0% increase in average gift

Key Learnings

With 100% level of confidence, we experienced a +803% increase in donor conversion rate with this treatment.

This conversion rate increase translated into an increase of revenue by +783% (also with a 100% level of confidence). We also noted a +1002% increase in ad clickthrough rate, as well.

It would appear that promoting an offer instead of membership is more effective at driving conversions.

Our thought is that this is because promoting membership to existing members was met with the outcome of no clicks because people thought: “I’m already a member, so I don’t need to click to activate my membership.” By promoting an offer instead, we capitalized upon this dormant audience and the instant donation page behind the offer download page provided an opportunity for additional giving to people who have not be clicking Dear Reader ads in the past.


Experiment Documented by...

Greg Colunga

Greg is Executive Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.