How leading with a question impact email click rate
Alliance Defending Freedom
Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.
Experiment Summary
Timeframe: 07/19/2019 - 07/23/2019
Alliance Defending Freedom was sending out one of their weekly cultivation emails as part of a longitudinal experiment to improve email engagement. In the previous week, we had observed a surprising result in an experiment. When starting the email with a personal anecdote, we saw a decreased level of engagement. We had a hypothesis that this may be due to the way the two emails differed in their introductions. The control started with the question to the reader while the treatment started with the story.
Since there were several other factors that could have led to the observed result, we decided to isolate the question factor and run another A/B split. The only difference in the control and treatments email would then be whether they begin with a question or a statement.
Research Question
Does leading an email with a question increase engagement?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Opening Statement | 14.9% | ||
| T1: | Opening Question | 16.0% | 7.3% | 89.8% |
This experiment has a required sample size of 8,558 in order to be valid. Since the experiment had a total sample size of 11,931, and the level of confidence is not above 95% the experiment results are not valid.
Key Learnings
While we did not reach a statistical level of confidence, we were quickly approaching a validation with a 90% level of confidence. This would suggest that leading with a question is a better way to capture visitor attention that the standard introduction. However, we would need to test this further to validate the concept.