How simplifying your ad impacts donor conversion rate
Americans for Prosperity
Experiment Summary
Ended On: 03/17/2021
As a part of an open letter (or petition) promotion campaign, Americans for Prosperity wanted to experiment with simplifying an otherwise verbose and complicated open letter promotion.
They thought that by presenting a simple “Yes” or “No” question to the person the ad was being served to, that it could improve clickthrough and email signup rates, subsequently positively impacting donor conversion and revenue for the same dollar spent to advertise the campaign.
Research Question
We believe that simplifying the ad experience for prospective donors will achieve an increase in donor conversion rate.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.08% | ||
| T1: | Treatment #1 | 0.13% | 64.4% | 96.2% |
This experiment has a required sample size of 32,200 in order to be valid. Since the experiment had a total sample size of 68,653, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 64.4% increase in conversion rate
× 0% increase in average gift
Key Learnings
With a 96.2% level of confidence, we observed an increase in donor conversion by +64.4% by simplifying the ad experience in such a way that people would be presented with a simple “Yes” or “No” answer to a question, and would be called to take action today if they aligned with the organization (in this case “No”).
In fact, we saw increases at every step of the funnel.
Here is the lift that we achieved across the treatment version of the campaign:
- Ad CTR: +268.8% (LoC: 100%)
- Email Signup: +142.4% (LoC: 100%)
- Donor Conversion Rate: +64.4% (LoC: 96.2%)
- Revenue: +52.7% (LoC: 98%)
The results achieved with the simplified approach to the ad experience really started by dramatically decreasing the cost per click — showing a -80% decrease in cost per click while nearly quadrupling the number of clicks we saw against the control experience.
This higher volume of clicks for the same expense delivered similar more cost-effective results all of the way throughout the process, which boosted ROI by 100%, and decreased the cost to acquire an email by -69% and the cost to acquire a donor by more than -40%.
There was one more observation worth mentioning, as well. This approach had such an improvement in post engagement that it contributed to the lowering of costs for the outcomes we received. The question/answer-based approach created shares and comments that put this ad extended the reach of these campaigns. Specifically, we observed an 88% increase in post comments (LoC: 100%) due to this approach.
Where possible, experimenting with a very simple approach to issues related to a reader’s beliefs in the form of a “Yes” or “No” question can potentially provide dramatic increases in clicks that the efficiency metrics on cost per outcome drop to levels not seen before.
It should be noted that Facebook has rejected some ad images that (like above) look like they have buttons on them. So, further experimentation may be required to use banners or something similar when/if future ads like this are rejected.