How leading with a question impact email click rate Experiment ID: #17108

Alliance Defending Freedom

Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.

Experiment Summary

Timeframe: 7/19/2019 - 7/23/2019

Alliance Defending Freedom was sending out one of their weekly cultivation emails as part of a longitudinal experiment to improve email engagement. In the previous week, we had observed a surprising result in an experiment. When starting the email with a personal anecdote, we saw a decreased level of engagement. We had a hypothesis that this may be due to the way the two emails differed in their introductions. The control started with the question to the reader while the treatment started with the story.

Since there were several other factors that could have led to the observed result, we decided to isolate the question factor and run another A/B split. The only difference in the control and treatments email would then be whether they begin with a question or a statement.

Research Question

Does leading an email with a question increase engagement?


C: Opening Statement
T1: Opening Question


Treatment Name Click Rate Relative Difference Confidence
C: Opening Statement 14.9%
T1: Opening Question 16.0% 7.3% 89.8%

This experiment has a required sample size of 8,558 in order to be valid. Since the experiment had a total sample size of 11,931, and the level of confidence is not above 95% the experiment results are not valid.

Key Learnings

While we did not reach a statistical level of confidence, we were quickly approaching a validation with a 90% level of confidence. This would suggest that leading with a question is a better way to capture visitor attention that the standard introduction. However, we would need to test this further to validate the concept.

Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.