How targeting individual states for a national campaign impacted donor conversion rate
Americans for Prosperity
Experiment Summary
Ended On: 05/13/2020
For a petition being run by AFP, we wanted to determine if utilizing state-specific messaging would be a better motivator for audiences. We selected two key states that had sizable audiences and had grassroots efforts which we could utilize in the advertising messaging. These states were Texas and Florida.
Research Question
Would breaking out FL and TX audiences and talking to them specifically impact donor conversion rate or revenue?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.09% | ||
| T1: | Treatment #1 | 0.05% | -39.8% | 99.8% |
This experiment has a required sample size of 42,216 in order to be valid. Since the experiment had a total sample size of 209,977, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 39.8% decrease in conversion rate
× 0% increase in average gift
Key Learnings
After combining the state-specific data collected for the FL and TX campaigns and comparing it to the results of the national campaign, we found that with a 99.7% level of confidence, that this approach decreased donor conversion rate by 39.8%.
Other metrics impacted were:
- Ad Clicks: +19.8% (LoC: 100%)
- Emails Acquired: +19.9% (LoC: 100%)
- Revenue: -52.9% (LoC: 100%)
It would appear that the state-specific ad messaging and landing pages increased ad clicks and email signups, but did not scale through donor conversion rate and revenue decreased even more.
Key Takeaway:
This doesn’t mean that state-specific language and messaging won’t work, but rather that when there is a nation-wide message that is resonating, applying geographical restrictions to the ad service platform’s targeting will limit it’s ability to serve the “right ad, to the right person, at the right time” and produce the best cost per outcome.
Further testing may be required, but it might be advisable to not limit by geography with national campaigns that are working well in the future.