Using the word “pro-life” in a subject line vs. mystery
Care Net
Experiment Summary
Ended On: 05/13/2019
This subject line test was performed in one of our weekly “weekend update” emails.
Research Question
We wanted to see if using the word “pro-life response” would increase open rates, even against a subject line with mystery.
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | What Essence magazine said about pregnancy centers... | 14.9% | ||
| T1: | A pro-life response to Essence magazine | 15.8% | 6.6% | 99.9% |
This experiment has a required sample size of 10,323 in order to be valid. Since the experiment had a total sample size of 64,252, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
6.6% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The answer is yes, it did increase the open rate, but not by very much. It would be interesting to test this again in a test where the subject lines are more similar.