How asking a question about Christmas gifts impacted clickthrough rates.
CRISTA Ministries
Experiment Summary
Timeframe: 11/07/2018 - 11/21/2018
Most of the messaging for the Global Gift Guide catalog has always taken a very transactional tone historically. When “asking” instead of “telling”, we wanted to test what, if any, impact would be had related to open or clickthrough rates.
Research Question
Would engaging the housefile with questions improve email engagement rates, rather than consistently telling them what to do regarding shopping the Global Gift Guide?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | 3 reasons to give a life-changing gift | 0.21% | ||
| T1: | Why do you give Christmas gifts? | 0.38% | 83.7% | 99.9% |
This experiment has a required sample size of 7,555 in order to be valid. Since the experiment had a total sample size of 41,906, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
83.7% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
With a 99.8% confidence score, it appears that using questions to draw the audience in to read/consider Global Gift Guide gift options will create a higher level of “consideration”, which will in turn create greater clickthrough rates.