How framing offer content in a Facebook ad impacts email acquisition Experiment ID: #7738

Dallas Theological Seminary

The DTS mission is, “to glorify God by equipping godly servant-leaders for the proclamation of His Word and the building up of the body of Christ worldwide.” They strive to help men and women fulfill the Great Commission and the Great Commandment, or more simply: Teach Truth. Love Well.

Experiment Summary

Timeframe: 9/27/2017 - 10/27/2017

Dallas Theological Seminary was promoting their online course Revelation when they noticed an opportunity to test. In their original Facebook course promotion ad, they called on viewers to ‘study with one of DTS’ renowned professors, for free.’ Through previous message testing, they’ve seen that this is a strong value proposition for a lot of their online resources and material. However, they wondered: Will a stronger focus on the book being studied (i.e. Revelation) and its complexity increase motivation and email acquisition rate?

They launched an A/B test to find out.

Research Question

Will a Facebook ad that frames the offer content adequately increase email acquisition?

Design

C: Control
T1: Treatment 1

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Control 0.53%
T1: Treatment 1 1.3% 143.0% 100.0%

This experiment has a required sample size of 1,539 in order to be valid. Since the experiment had a total sample size of 30,799, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 143.0% increase in conversion rate
× 0% increase in average gift

Key Learnings

Dallas Theological Seminary found that their treatment version did increase email acquisition by 143%. It’s clear that offering more context to the book being studied in the course increased motivation of viewers on Facebook and drove far more course signups then the control version. While the original value proposition remains strong, it was much shorter in length.

This has prompted further message testing that focuses on individual course content, as well as the true value of learning from actual seminary professors for free.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Grant Klembara

Grant is an Optimization Director. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.