How utilizing statistics in an email appeal influenced results
Buckner International
Buckner International is a global ministry dedicated to the transformation and restoration of the lives we serve. We are a Christ-centered organization that delivers redemptive ministry to the most vulnerable from the beginning to the ending of life.
Experiment Summary
Ended On: 12/27/2018
Buckner International was sending out their 2018 calendar year end campaign focused on the impact and need behind foster care and adoption. When discussing as a team, there was debate surrounding the use of statistics in the email appeals. There was published research indicating that statistics in an appeal could potentially lower response rates. We decided to test this theory with an email during the year end campaign.
Research Question
Will the use of statistics influence the results of an email appeal?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | No Statistics | 0.25% | ||
| T1: | With Statistics | 0.24% | -2.7% | 4.1% |
This experiment has a required sample size of 4,124,939 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
We found that there was no statistically significant difference between either of the treatments with regards to both donations and clicks. It is worth noting that this is just one attempt at validating this theory. We would need to test it a few more times to truly understand the impact of incorporating statistics into our appeals.