How the value prop of a subject line affects open rate in an appeal
Buckner International
Buckner International is a global ministry dedicated to the transformation and restoration of the lives we serve. We are a Christ-centered organization that delivers redemptive ministry to the most vulnerable from the beginning to the ending of life.
Experiment Summary
Ended On: 05/01/2018
Buckner International was sending out their April foster camp scholarship appeal, and wanted to test the subject line to try to increase open rates.
Research Question
Does a camp-focused subject line versus a more vague, opportunity-focused subject line increase open rates in an email?
Design
Results
| Treatment Name | Open Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | A unique opportunity | 9.9% | ||
| T1: | A different kind of camp | 7.8% | -20.9% | 100.0% |
This experiment has a required sample size of 1,456 in order to be valid. Since the experiment had a total sample size of 79,436, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
20.9% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The treatment subject line decreased opens by 21%. This could be due to a lessened sense of mystery than the original subject line, or that the treatment subject line otherwise did not motivate people to open the email.