How a text ad affects conversion for a free trial offer signup Experiment ID: #9358

EWTN

Experiment Summary

Ended On: 8/21/2018

National Catholic Register had run many animated banner ads throughout their site to encourage online readers to sign up for a free three-month trial. While the animated banners were eye-catching and featured well-known Catholic personalities, they wondered if they required the viewer to sit for a long period in front of the ad to see the payoff—and if those viewers, once the reveal was complete, missed the CTA entirely. Also, they wondered if the narrative form of the ad was less clear and straightforward than a simple text offer with a bold call-to-action.

They created a treatment ad, run through their ad server, that presented the trial offer in a clear way with a big CTA button. They evenly apportioned the ads to run to NCRegister viewers and closely tracked metrics to determine clicks and conversions.

Research Question

How will a text-only ad perform when compared to an animated banner?

Design

C: Animated Ad
T1: Text-based Ad

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Animated Ad 0.01%
T1: Text-based Ad 0.02% 296.0% 100.0%

This experiment has a required sample size of 42,415 in order to be valid. Since the experiment had a total sample size of 310,791, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 296.0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The text ad produced a 296% lift in overall conversions, from impressions to clicks, nearly quadrupling the number of conversions from the animated banner. Interestingly, the text banner reduced clickthrough rate by 18.5%, but increased follow-on conversion by 386%. It was hypothesized that the primary factor in the difference in clicks was due to the much larger clickable area on the animated banner, which, on mobile, was much more susceptible to accidental clicks. However, the conversion rate from the text ad was so much higher that some portion of this conversion increase can also be attributed to the increased clarity of the offer—greater understanding from the end user resulted in a “more motivated click”.

This produced future test ideas, like adding a picture of the print edition, and specifically calling the offer a print edition.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Jeff Giddens

Jeff is the President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.