How optimizing for “Subscribers” impacts recurring donor acquisition Experiment ID: #21563

EWTN

Experiment Summary

Timeframe: 5/13/2020 - 6/10/2020

Roughly a year ago, EWTN validated that optimizing their Facebook ads for “Purchases” yielded significantly more donors than optimizing for ”Custom Conversions” (aka emails) or even for optimizing for “Donors” as Facebook defines them. Our hypothesis for this result is that Facebook’s advertising algorithm incorporated online buying behavior with ”Purchase” optimization that the other metrics did not. Ultimately, this experiment drastically improved acquisition efforts and significantly lowered costs.

Recently, Facebook rolled out additional conversion options that are available for optimization. One of those options is a ”Subscription” which is intended to represent an ongoing monthly payment. Given EWTN’s emphasis on acquiring recurring donors, we decided to run an experiment to see if this metric could acquire more recurring donors than “Purchase” optimization. It should be noted that the campaign we ran this experiment on did not have a specific recurring ask but did include the option of giving a recurring gift.

Research Question

Will optimizing Facebook ads for “Subscriptions” increase recurring donors when compared to “Purchases?”

Design

C: Purchase Optimization
T1: Subscription Optimization

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Purchase Optimization 0.01%
T1: Subscription Optimization 0.01% -41.6% 75.9%

This experiment has a required sample size of 291,548 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

After running for nearly a month, the Subscriber optimization reduced recurring donor acquisition by 41.6% with a 75.9% level of confidence. This means it was not a statistically significant learning but, after a month of running, it was also not likely that it would reverse course and become a statistically significant lift to recurring donor acquisition. Given this fact, the decision was made to end the experiment before we reached a statistically significant result.

When analyzing the other metrics, we discovered that there were several other notable trends:

  • The Subscriber optimization also decreased overall donor conversion rate by 12.8% with a 68% level of confidence (so it was not a statistically valid decrease.)
  • Decreased immediate overall revenue by 37%.

Based upon all of this, it would appear that the Subscriber optimization is not a good way to improve recurring donor acquisition.


Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.