How Making a Statement in an email Subject Line impacted open rates
Care Net
Experiment Summary
Ended On: 05/16/2018
With our weekly weekend update email, we are always working to increase engagement rates. We were wondering if making a statement in the subject line would have better response than asking a question.
Research Question
Does a statement in the subject line of an email perform better than a question.
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Question-based subject line | 14.7% | ||
| T1: | Statement based subject line | 15.8% | 7.6% | 100.0% |
This experiment has a required sample size of 8,004 in order to be valid. Since the experiment had a total sample size of 54,917, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
7.6% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
People were more motivated to open an email with the promise of an answer to the question “should men shut up about abortion” than with the question itself. In addition to increasing the open rate, the click-through-rate increased 31.87%!
Continued testing in this area will help us see the full extent of this learning.