How asking a specific question in an ad impacts clickthrough rate
Texas Public Policy Foundation
Experiment Summary
Ended On: 04/27/2021
As part of a direct ask campaign for monthly donations, the Texas Public Policy Foundation wanted to test if specific policy ads would outperform a general invitation to join a branded recurring donor program. They had seen how other organizations had seen increases in traffic conversion through simplifying ads and wanted to test the concept on their images for the policy-specific ads.
Research Question
We believe that simplified Facebook ads posing a relevant question for potential donors will achieve an increase in traffic to the donation page.
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 2.9% | ||
| T1: | Treatment #1 | 38.8% | 1,223.4% | 100.0% |
This experiment has a required sample size of 8 in order to be valid. Since the experiment had a total sample size of 276,482, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
1,223.4% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
With a 100% level of confidence, we observed a 1,223.4% lift in click rate on these Facebook ads that included a banner with the option to select “Yes” or “No” to indicate if they align with the organization’s stance on specific policy areas. Once the visitors clicked on the call to take action, we were able to present them with the recurring donation ask to support TPPF’s efforts in that specific policy area they just told us they have a shared belief.
What’s interesting is that when we looked at our donor conversion rate on the donation page, we actually saw an 80.5% decrease with 100% Level of Confidence—which means that now that we’ve determined how to increase the number of clicks in order to drive a substantial amount traffic to the page, we can now shift our focus to running experiments to improve conversion on the donation page.
While the increase in traffic is not necessarily producing visits from highly motivated donors to the page, we still see this as a worthwhile effort because once they visit the donation page we have the ability to set up additional campaigns designed to retarget them and find other opportunities to reach them distinctly with content related to their policy interests.