Increasing cultivation email engagement by asking readers to align with a statement of beliefs Experiment ID: #20375

Back to the Bible

Experiment Summary

Timeframe: 12/17/2019 - 12/31/2019

In the middle of Back to the Bible’s year-end campaign, they included a cultivation email with no ask to serve as an engagement piece for their audience. We wanted to test whether asking their audience to align with their statement of beliefs would receive higher engagement than a general feedback survey.

Research Question

Would asking the reader to confirm their shared values increase email engagement?

Design

C: Version A - Value Prop Survey
T1: Version B - Statement of Beliefs

Results

Treatment Name Click Rate Relative Difference Confidence
C: Version A - Value Prop Survey 12.2%
T1: Version B - Statement of Beliefs 19.8% 62.4% 100.0%

This experiment has a required sample size of 178 in order to be valid. Since the experiment had a total sample size of 2,667, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    62.4% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

We consistently found that across all email segments—including those who were new to the file (increased click rate by 39.7%), often engaged (recorded in this experiment), sometimes engaged (increased click rate by 99.2%) , and rarely engaged (increased click rate by 50.9%)— that the statement of beliefs outperformed the value prop survey. Not only were the open and click through rates higher on the statement of beliefs email, increasing traffic to the page, but we also saw an overall 64.57% conversion rate on the statement of beliefs page compared to a 41.92% conversion rate on the general feedback survey. These higher response rates indicate increased investment from the readers. In future testing, we will evaluate how this affects downstream donor conversion.


Experiment Documented by...

Rebekah Josefy

Rebekah is Optimization Associate. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.