How the messaging and sender in a year-end email impacts donor conversion Experiment ID: #19472

EWTN

Experiment Summary

Timeframe: 12/19/2019 - 12/19/2019

During EWTN’s year-end campaign, they wanted to test the impact on a different sender for one of their appeals. The appeals typically come from the CEO of the organization, Michael Warsaw. The control email came from Michael with messaging around what EWTN wanted/needed to accomplish in 2020. He communicated specific goals and initiatives in the copy and had a more serious tone to it. We tested this approach against an email from the VP of Mission Advancement, Scott Brown. This treatment email also highlighted goals for 2020 in it as well, but took a more conversational/lighter tone to it.

Research Question

Would a different tone and sender increase donor conversion?

Design

C: Michael P. Warsaw as sender
T1: Scott F. Brown as sender

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Michael P. Warsaw as sender 0.06%
T1: Scott F. Brown as sender 0.06% -8.1% 35.7%

This experiment has a required sample size of 1,746,352 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

We learned that the tone and sender of this email had no impact on donor conversion. While the treatment proved to not increase donor conversion, it also showed us that a different sender – other than the organization’s CEO – could be the sender of their email appeals and have no NEGATIVE impact on conversion. This is a great learning and something we will test again outside of a high-urgency campaign.


Experiment Documented by...

Courtney Gaines

Courtney is a Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.