Increasing cultivation email engagement by asking readers to align with a statement of beliefs
Back to the Bible
Experiment Summary
Timeframe: 12/17/2019 - 12/31/2019
In the middle of Back to the Bible’s year-end campaign, they included a cultivation email with no ask to serve as an engagement piece for their audience. We wanted to test whether asking their audience to align with their statement of beliefs would receive higher engagement than a general feedback survey.
Research Question
Would asking the reader to confirm their shared values increase email engagement?
Design
Results
Treatment Name | Click Rate | Relative Difference | Confidence | |
---|---|---|---|---|
C: | Version A - Value Prop Survey | 12.2% | ||
T1: | Version B - Statement of Beliefs | 19.8% | 62.4% | 100.0% |
This experiment has a required sample size of 178 in order to be valid. Since the experiment had a total sample size of 2,667, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
62.4% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
We consistently found that across all email segments—including those who were new to the file (increased click rate by 39.7%), often engaged (recorded in this experiment), sometimes engaged (increased click rate by 99.2%) , and rarely engaged (increased click rate by 50.9%)— that the statement of beliefs outperformed the value prop survey. Not only were the open and click through rates higher on the statement of beliefs email, increasing traffic to the page, but we also saw an overall 64.57% conversion rate on the statement of beliefs page compared to a 41.92% conversion rate on the general feedback survey. These higher response rates indicate increased investment from the readers. In future testing, we will evaluate how this affects downstream donor conversion.
Question about experiment #20375
If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.