How SpotRight compares to other Facebook Audience Models for prospecting Experiment ID: #20182

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Ended On: 3/5/2020

The Heritage Foundation wanted to experiment with using a modeled audience from the new SpotRight tool available from Wiland, one of their data partners. SpotRight claims to use a proprietary model to produce a higher prospective customer list that is 10x more likely to become a buyer.

In an attempt to validate the audiences it built, we setup an experiment with a Heritage modeled audience based upon targeting keywords and pushed approximately 20 million people into a targetable audience on Facebook. We then gave the campaign a modest budget (about 5% for the quarter) and a 19-day timeline to run our existing prospecting ads.

Research Question

How does SpotRight’s audience compare to other audiences used in prospecting and targeting for new donor acquisition?

Design

C: Control
T1: Treatment #1
T2: Treatment #2
T3: Treatment #3

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Control 0.01%
T1: Treatment #1 0.01% 157.5% 99.9%
T2: Treatment #2 0.01% 141.0% 99.0%
T3: Treatment #3 0.01% 35.5% 70.0%

This experiment has a required sample size of 164,816 in order to be valid. Since the experiment had a total sample size of 1,342,906, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 157.5% increase in conversion rate
× 0% increase in average gift

Key Learnings

In reviewing the results from the targeting experiment, we can conclusively say that the SpotRight modeled audience increased donor conversion rate over the “control” (which is targeting fans of similar causes and/or interest-based audiences) by 141%.

We also noticed that the SpotRight modeled audience performed as well as the Facebook lookalike modeled audiences. What does this mean? It means that SpotRight may be a good starting point when launching a new brand, program or fundraising initiative if you do not have the minimum number of donors to model an audience from.

Because Facebook audiences should really be thought of as “overlapping circles on a Venn Diagram” that often overlap when you build multiple audiences that you are targeting, we thought that many of our known audiences (e.g. donors and/or subscribers) may have found their way into the SpotRight audience model.

So, we checked the Analytics on the visitors, and we found that 85% of all SpotRight visitors to our donation pages actually were first time visitors to our website, and when they converted, their average gift was only slightly less (-7%) than return visitors that came through the SpotRight campaign.

Where do we go from here? We wonder how this audience can scale moving forward, so we’ve quadrupled the budget to monitor how the SpotRight audience scales over a longer period of time (31 days) with more budget from the original experiment time frame.


Experiment Documented by...

Greg Colunga

Greg is Executive Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.