Dallas Theological Seminary

How subject line copy that “teases” affects open rate

Experiment ID: #2675

Dallas Theological Seminary

The DTS mission is, “to glorify God by equipping godly servant-leaders for the proclamation of His Word and the building up of the body of Christ worldwide.” They strive to help men and women fulfill the Great Commission and the Great Commandment, or more simply: Teach Truth. Love Well.

Experiment Summary

Timeframe: 11/17/2015 - 11/20/2015

Dallas Theological Seminary launched a free online course studying the Gospel of John. They were promoting it by renting email lists, and wanted to determine which subject lines encouraged more recipients to open the email. They developed two subject lines based on hypotheses about the prospect. The first was very direct: “Study the Gospel of John with DTS”. The second teased the offer, and expanded from the acronym: “A special offer from Dallas Theological Seminary”.

Research Question

Which subject line results in a higher open rate?

Design

C: Control
T1: Treatment 1

Results

 Treatment NameOpen RateRelative DifferenceConfidence
C: Control 6.7%
T1: Treatment 1 9.2%37.3% 100.0%

This experiment has a required sample size of 898 in order to be valid. Since the experiment had a total sample size of 67,297, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    37.3% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

This tells us that prospects are more likely to open an email when we tease the contents. While clarity has proven to trump persuasion over and over again, the subject line is a different element. It only has one job — to win the open. We can’t get people to click if they don’t open. So this suggests that teasing recipients of this email might increase open rate, which is the metric that affects every other following metric.


Experiment Documented by Jeff Giddens
Jeff Giddens is President of NextAfter.

Question about experiment #2675

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.