How a custom Facebook conversion event to target recurring purchases impacted recurring donor conversion rate | NextAfter
The Heritage Foundation

How a custom Facebook conversion event to target recurring purchases impacted recurring donor conversion rate

Experiment ID: #37179

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Ended On:

As a part of experimenting with Facebook’s custom conversion events, we wanted to experiment with whether the custom-defined conversion event which specifically identified and send payment details back to Facebook’s algorithm to differentiate between a single gift and a monthly recurring gift purchase type would impact the recurring donor conversion rate.

For this campaign, we partnered with Wiland to identify a custom audience of lookalike donors who are most likely (per the Wiland model) to become a recurring donor for The Heritage Foundation.

The only variable in this experiment was (a) the standard, out of the box, “purchase” event from Facebook vs. (b) the custom-defined “recurring purchase” event that we created and installed in Facebook.

Research Question

Would a custom “recurring purchase” event type increase a campaign’s recurring donor conversion rate?

Design

C: Standard "Purchase" Event
T1: Custom "Recurring Purchase" Event

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Standard "Purchase" Event 0.02%
T1: Custom "Recurring Purchase" Event 0.01% -63.9% 99.5%

This experiment has a required sample size of 55,328 in order to be valid. Since the experiment had a total sample size of 252,098, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 63.9% decrease in conversion rate
× 0% increase in average gift

Key Learnings

With a 99.5% level of confidence, we observed a significant decrease of -63.9% in recurring donor conversion rate. We observed a -62% decrease in revenue (essentially, a proportionate decrease relative to the lesser quantity of donors coming through the treatment experience) with a 100% level of confidence, as well.

It should be noted that we turned this experiment off only about 72% of the way through the minimum number of transactions we would like to see in a donor conversion study of this type (and even less—about 35%—of a revenue lift study), but the volume was still high enough and the spread great enough between the control and treatment experiences that we believed it would be best to not finish the study with the target minimum number of gifts we would normally like to see.

Further experimentation may be required with the purchase event programming and audience targeting combinations, but this certainly seems to point towards a decrease in donor conversion for recurring donor-focused campaigns.


Experiment Documented by Greg Colunga
Greg Colunga is Executive Vice President at NextAfter.

Question about experiment #37179

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.