How subject line personalization affects open rate
Dallas Theological Seminary
The DTS mission is, “to glorify God by equipping godly servant-leaders for the proclamation of His Word and the building up of the body of Christ worldwide.” They strive to help men and women fulfill the Great Commission and the Great Commandment, or more simply: Teach Truth. Love Well.
Experiment Summary
Timeframe: 05/04/2016 - 06/15/2016
Dallas Theological Seminary was promoting their annual Christmas devotional series to their house email file. In the past, they had used a very straightforward subject line to promote the offer: Devotionals for the Christmas Season. But their new marketing platform allowed them to personalize the subject line with the recipient’s firstname, so they put it to a test.
They sent 10% of their house file each treatment and collected data to determine a winner.
Research Question
How does subject line personalization affect open rate?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 13.6% | ||
| T1: | Treatment 1 | 16.6% | 22.5% | 100.0% |
This experiment has a required sample size of 1,055 in order to be valid. Since the experiment had a total sample size of 12,746, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
22.5% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The treatment with the personalized subject line produced a 22.5% lift in open rate, which showed that including the recipient’s first name in the subject line increases open rate.
But the personalization wasn’t the only factor—the treatment subject line also obscured the actual payoff of the email by not disclosing the offer. This appeared to heighten interest in what the “reveal” would be, intensified by the personalization factor.
The combination of these two factors led to a significant increase in open rate, which helped inform future subject line copy. However, the “trick” of personalization also may come with a significant diminishing return if overused.