How personalizing the Working Preacher fall campaign email effected the donor conversion rate
Luther Seminary
Experiment Summary
Ended On: 11/17/2021
This experiment was run on the first email sent to the housefile for the Working Preacher fall campaign.
The goal of this experiment was to increase donor conversion rate by personalizing the verbiage and simplifying the design of the email.
Within the treatment, we changed multiple factors including:
- Beginning the email with a personalized story of how Working Preacher has positively affected a preacher’s job and messages
- Explaining the why behind how coming up with new and interesting content, week-after-week, can be very difficult and strenuous
- Simplifying the design and removing the templated “look” of the email to seem like it’s coming from a real person
- Changing the wording from (in the control) speaking from an organization to (in the treatment) speaking in a conversation
- Putting more emphasis on the donor and how important their donation is to this campaign
- Adding more hyperlinks throughout the whole email to keep giving a donation at the top of mind for the potential donor
Research Question
We believe that personalizing an email for potential donors will achieve a higher donor conversion rate.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 1.6% | ||
| T1: | Treatment #1 | 1.5% | -5.6% | 21.6% |
This experiment has a required sample size of 146,100 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
We found that personalizing an email that is asking for donations to the Working Preacher fall campaign had no effect on the donor conversion rate.
We believe that Working Preacher’s donors are already accustomed to how Working Preacher talks throughout their campaigns, so personalizing the email did not have an effect on the donor’s motivation to give.