How a more casual subject line impacts opens
Alliance Defending Freedom
Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.
Experiment Summary
Timeframe: 07/12/2019 - 07/16/2019
Alliance Defending Freedom was continuing the longitudinal experiment to increase cultivation to a targeted segment of their email house file. The goal of these emails was to simply engage the audience and provide them with ongoing content and stewardship. With that in mind, we wanted to find ways to increase that engagement.
Since these emails were intended to appear as if they were one-to-one communications, we wanted to see if a more casual subject line could impact the open rate. This would be compared the to more straight-forward subject lines often used.
Research Question
Will a more casual subject line increase email open rate?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 19.9% | ||
| T1: | Treatment #1 | 21.5% | 8.1% | 100.0% |
This experiment has a required sample size of 4,904 in order to be valid. Since the experiment had a total sample size of 61,123, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
8.1% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
With the more casual “I can’t even imagine…” subject line, we saw an 8.1% increase to email opens. This subject line had two key differences from the control:
- It created a mystery that the reader would only solve by opening the email
- It was far more personal using the “I” in the subject
To know which of these factors had the biggest impact, we would need to isolate them individually in a future experiment.