How asking a specific question in an ad impacts clickthrough rate - NextAfter
Texas Public Policy Foundation

How asking a specific question in an ad impacts clickthrough rate

Experiment ID: #58652

Texas Public Policy Foundation

Experiment Summary

Ended On: 04/27/2021

As part of a direct ask campaign for monthly donations, the Texas Public Policy Foundation wanted to test if specific policy ads would outperform a general invitation to join a branded recurring donor program. They had seen how other organizations had seen increases in traffic conversion through simplifying ads and wanted to test the concept on their images for the policy-specific ads.

Research Question

We believe that simplified Facebook ads posing a relevant question for potential donors will achieve an increase in traffic to the donation page.

Design

C: Control
T1: Treatment #1

Results

  Treatment Name Click Rate Relative Difference Confidence
C: Control 2.9%
T1: Treatment #1 38.8% 1,223.4% 100.0%

This experiment has a required sample size of 8 in order to be valid. Since the experiment had a total sample size of 276,482, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    1,223.4% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

With a 100% level of confidence, we observed a 1,223.4% lift in click rate on these Facebook ads that included a banner with the option to select “Yes” or “No” to indicate if they align with the organization’s stance on specific policy areas. Once the visitors clicked on the call to take action, we were able to present them with the recurring donation ask to support TPPF’s efforts in that specific policy area they just told us they have a shared belief.

What’s interesting is that when we looked at our donor conversion rate on the donation page, we actually saw an 80.5% decrease with 100% Level of Confidence—which means that now that we’ve determined how to increase the number of clicks in order to drive a substantial amount traffic to the page, we can now shift our focus to running experiments to improve conversion on the donation page.

While the increase in traffic is not necessarily producing visits from highly motivated donors to the page, we still see this as a worthwhile effort because once they visit the donation page we have the ability to set up additional campaigns designed to retarget them and find other opportunities to reach them distinctly with content related to their policy interests.


Experiment Documented by Rebekah Josefy
Rebekah Josefy is an Optimization Director at NextAfter.

Question about experiment #58652

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.