Dallas Theological Seminary

How clarity in the call-to-action affects email acquisition

Experiment ID: #6177

Dallas Theological Seminary

The DTS mission is, “to glorify God by equipping godly servant-leaders for the proclamation of His Word and the building up of the body of Christ worldwide.” They strive to help men and women fulfill the Great Commission and the Great Commandment, or more simply: Teach Truth. Love Well.

Experiment Summary

Timeframe: 01/25/2017 - 02/19/2017

Dallas Theological Seminary launched a new course called “Can You Trust the Bible?” They had shot a video of their Academic Dean explaining the course, and wanted to see how it performed against a direct invitation from the President to join the course. So they created two entrance paths to the course. The first was their control—a direct ask, via email, from the President of DTS. The second was an invitation from the President with a call-to-action to “watch the video to learn more about the course”. They hypothesized that the softer call-to-action would drive more traffic to the landing page, and the short video with a clear call-to-action would produce more overall conversions.

They split their file and closely monitored the data to determine a winner.

Research Question

Will a softer call-to-action with a compelling video increase course signups?

Design

C: Direct ask
T1: Video ask

Results

 Treatment NameConv. RateRelative DifferenceConfidence
C: Direct ask 3.9%
T1: Video ask 2.7%-32.6% 100.0%

This experiment has a required sample size of 1,485 in order to be valid. Since the experiment had a total sample size of 22,129, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 32.6% decrease in conversion rate
× 0% increase in average gift

Key Learnings

The treatment produced a 32.6% decrease in signups from the control. This shows that the softer ask does not produce more motivation to sign up for the course, even with the added benefit of the video. There could be a few factors at play here. First, all clicks are not created equal—a click to “learn more”, in this case, is far less valuable than a click to “sign up”. Second, video might actually be obscuring the value proposition by not allowing the user to read it in their own time. Many people can read much faster than a video can communicate information, which leads them to be more likely to accept an offer where they control the cadence of the value proposition.

This does not necessarily mean, however, that the video doesn’t have a place in the marketing schedule of a new course—it may just mean that it needs to be placed later in the schedule to motivate users who were not incentivized by the original message.


Experiment Documented by Jeff Giddens
Jeff Giddens is President of NextAfter.

Question about experiment #6177

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.