How message tone of an email affects response rate - NextAfter
Dallas Theological Seminary

How message tone of an email affects response rate

Experiment ID: #893

Dallas Theological Seminary

The DTS mission is, “to glorify God by equipping godly servant-leaders for the proclamation of His Word and the building up of the body of Christ worldwide.” They strive to help men and women fulfill the Great Commission and the Great Commandment, or more simply: Teach Truth. Love Well.

Experiment Summary

Timeframe: 01/21/2015 - 01/22/2015

As  part of their new monthly email appeal strategy, Hillsdale College wanted to learn which messaging tone produced the best response rate.

Research Question

How does messaging tone affect donation response rate in an email appeal?

Design

C: Reverse the Damage
T1: Fuel the Movement

Results

  Treatment Name Conv. Rate Relative Difference Confidence
C: Reverse the Damage 0.01%
T1: Fuel the Movement 0.02% 84.8% 93.0%

This experiment has a required sample size of 173,248 in order to be valid. Since the experiment had a total sample size of 295,912, and the level of confidence is not above 95% the experiment results are not valid.

Key Learnings

The experiment utilized two distinct emails and two distinct landing pages. In the experiment, the softer tone email generated 257% increase in click through rate. This increase in traffic illustrates a concept that we have observed in other experiments.  By reducing the “sharpness” of the ask in the email, we can increase the volume of traffic to the landing page, however the visitors will be less motivated causing the on-page conversion rate to be lower than the Control.   However, if the increase in traffic is significant enough, the net result will be more revenue.

The softer tone treatment delivered a response rate 85% higher than the original control with a 40% higher average gift.

This experiment is a great example of the importance of analyzing the entire conversion pathway, from email delivery to successful gift, when determining a winner.


Experiment Documented by NextAfter

Question about experiment #893

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.