Back to the Bible

How a more personal tone in a direct mail letter can impact donor conversion

Experiment ID: #21354

Back to the Bible

Experiment Summary

Timeframe: 3/3/2020 - 4/16/2020

Back to the Bible has shifted the tone and approach to their digital communication dramatically in the past year. The emails they send to people are very conversational, intentional, and personal. Their email file health has seen a dramatic improvement since this change was put in place. The focus of the emails has been to care for people and their spiritual life. Because of this responses digitally, we were curious how this approach might affect their Direct Mail strategy. To test this, we wrote two different Direct Mail letters. One letter was more traditional in nature – the call-to-action focused on the premium offer and included a P.S. at the end of the letter. The treatment was more conversational, soft, and personal in nature – more similar to the approach we’ve taken in their digital communication. We split their Direct Mail file in half and made sure there was an equal number of donors and nondonors for each variable.

Research Question

Will a more personal approach to Direct Mail impact donor conversion and overall revenue?

Design

C: Traditional Direct Mail
T1: More Personal Direct Mail

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: 3.2%
T1: More Personal Direct Mail 3.0% -4.5% 33.6%

This experiment has a required sample size of 113,858 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

After six weeks of the Direct Mail piece being in homes, we pulled the results of the experiment and found that there was no significant difference in donor conversion. We did see more overall revenue from the treatment, but this is mostly attributed to a handful of major gifts that came in from that variable. We will continue to monitor the results of this campaign over the next few weeks, but we’d also like to test this approach again and take an even more radical approach between the two letters. As you can see in the copy, the main body of the text was the same between the two versions.

Next, we’d like to test taking the digital appeal and just repurpose it for Direct Mail.


Experiment Documented by Courtney Gaines
Courtney Gaines is Vice President at NextAfter.

Question about experiment #6505

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.