How moving the problem to the front of the email affects donation rate
The Navigators
Experiment Summary
Ended On: 11/14/2023
The initial email of the Navigators fall Sustainers campaign used the example of how their team was able to go from “zero to one” to get missionaries on a specific college campus in Austin. However, this fact was buried in the email, which started with mission language that was somewhat organizational-centric.
The team hypothesized that the fact that a large college had ZERO missionaries on campus would be compelling and would engage more people if it was moved to the beginning of the email.
They believed by confronting the reader with this fact (and the fact that, thanks to donors, there now ARE missionaries on campus) that they would get more donations because more readers would understand the impact of their gift.
Research Question
We believe that highlighting the problem earlier in the email for email recipients will achieve an increase in conversion rate because more readers will understand the problem to which they can be the solution.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.00% | ||
| T1: | Treatment #1 | 0.01% | 449.3% | 98.7% |
This experiment has a required sample size of 64,105 in order to be valid. Since the experiment had a total sample size of 207,580, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 449.3% increase in conversion rate
× 0% increase in average gift
Key Learnings
The key learnings from this experiment are:
1. Positioning the problem earlier in the email was effective: By highlighting the fact that a large college had zero missionaries on campus and emphasizing the impact of donor support in solving this problem, the treatment email achieved a significant increase in donations. This suggests that readers are more likely to respond when confronted with a clear problem and understand their role in solving it.
2. Donors want to see the impact of their gift: The fact that the treatment email successfully conveyed the impact of donor support in addressing the problem suggests that donors want to know how their contributions are making a difference. Future experiments should focus on highlighting the tangible outcomes and benefits that donors can achieve through their support.
3. Consider the reader’s perspective: The original email was somewhat organizational-centric, starting with mission language. However, by repositioning the problem statement, the treatment email took into account the reader’s perspective and engaged them more effectively. Future experimentation should continue to prioritize the reader’s perspective and focus on conveying the message in a way that resonates with them.
Based on these learnings, future experimentation should continue to explore ways to highlight the problem and its impact early in the communication. Using donor-centered language and emphasizing the tangible outcomes of their support can help increase engagement and conversion rates. Additionally, ongoing testing should be conducted to fine-tune the messaging and ensure that it aligns with the readers’ motivations and interests.