How do the perceived benefits associated with becoming a recurring donor impact click-through rate?
The Navigators
Experiment Summary
Ended On: 02/27/2024
For The Navigator’s February Sustainers Appeal, we hypothesized that calling out the benefits of becoming a monthly partner would increase our click-through rate as well as our donor conversion rate. The control email emphasized the importance and need for monthly partners, focusing heavily on the difference it would make for the organization. Whereas, the treatment focused on the benefits for the donor themselves.
This test ran as a 50/50 split to their housefile audience in an effort to acquire more recurring donors as well as test what motivates their audience to give.
Research Question
We believe that including the benefits of becoming a monthly partner for the donor will achieve an increase in click through rate because of the perceived exclusivity and incentives associated with becoming a monthly partner.
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.79% | ||
| T1: | Treatment #1 | 0.85% | 7.4% | 52.7% |
This experiment has a required sample size of 181,958 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
The A/B experiment conducted for The Navigator’s February Sustainers Appeal tested the impact of emphasizing the benefits of becoming a monthly partner for the donor, as opposed to focusing on the organization’s need for monthly partners.
The treatment email, which highlighted the benefits for the donor, showed a 7.4% increase in Clicks for All Traffic compared to the control email. However, the statistical level of confidence was only 52.7%, meaning the results cannot be considered statistically valid.
However, the treatment greatly outperformed the control in terms of transactions. The treatment saw a 161.3% lift in transactions at a 99.6% level of confidence, meaning that the results are valid!
Therefore, the key learnings from this experiment are that The Navigators housefile is much more motivated to give when exclusivity and incentives are coupled with an appeal. While the results showed that readers were slightly more inclined to click the link when benefits to the donor were included, the individuals who did take the next step within the treatment, were much more motivated to give.