Luther Seminary

How a testimonial email from a donor performed against a faux-forward email during a fundraising appeal

Experiment ID: #195810

Luther Seminary

Experiment Summary

Ended On: 12/03/2024

During the critical calendar year-end fundraising period, Luther Seminary wanted to test the effectiveness of their Giving Tuesday PM email communication strategy. The traditional approach has used a faux-forward email, where the PM message includes a “forwarded” copy of the AM email, with a brief added note from the person forwarding it.
For this experiment, we tested the faux-forward strategy against a Treatment email that replaced the forwarded content with a testimonial from a real donor, Greg, sharing his personal reasons for supporting Luther Seminary. This test aimed to determine whether focusing on a donor testimonial would lead to higher engagement and more conversions compared to the repetitive messaging of the faux-forward email.
Donor stories would create stronger emotional connections and trust, motivating readers to act and contribute to Luther Seminary’s mission. A personal endorsement from a peer would resonate more deeply with recipients, leading to increased clicks to the donation page. At the same time, the faux-forward approach might contribute to disengagement or message fatigue.

Research Question

We believe that using a donor testimonial email from a real donor for people receiving a campaign appeal email will achieve an increase in email clicks to the donation page because personal stories can increase social proof and encourage readers to join them by becoming a donor.

Design

C: Control

T1: Treatment 1

Results

	Treatment Name	Conv. Rate	Relative Difference	Confidence
C:	Control	0.28%
T1:	Treatment 1	0.17%	-39.0%	98.0%

This experiment has a required sample size of 14,624 in order to be valid. Since the experiment had a total sample size of 41,203, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

0% increase in traffic
× 39.0% decrease in conversion rate
× 0% increase in average gift

Key Learnings

In this experiment, we found that the Treatment email, featuring a testimonial from a donor (Greg), led to a -39% decrease in clicks to the donation page, with a 97.9% statistical level of confidence. This result was unexpected, as we hypothesized that the personal donor story would increase trust and emotional connection, ultimately driving more clicks to the donation page. One major factor contributing to this decrease could be the -21.8% decrease in email opens for the Treatment email, with a 100% statistical level of confidence. This drop in opens suggests that the unfamiliar sender, Greg Myhr, may have impacted the email's ability to capture the recipient’s attention. People are more likely to open emails from familiar senders, a concept known as sender recognition or trust transfer. In this case, emails from Luther Seminary staff may have felt more familiar and trustworthy, prompting more recipients to open the email. Psychologically, cognitive fluency plays a role here: people are more likely to engage with content that feels easy, familiar, and comfortable. When the email is from a known and trusted source (like Luther Seminary staff), recipients may feel more at ease engaging with the content. When the email comes from an unfamiliar source (like a donor), it may subconsciously signal uncertainty or a potential risk, causing people to hesitate or overlook the email altogether. Additionally, while social proof can be a powerful motivator, it needs to be framed in a way that maximizes its impact. Although Greg’s testimonial aimed to leverage the principle of social proof by showing how a fellow donor supports Luther Seminary, the email’s unfamiliar sender might have diminished its credibility. In-group bias suggests that people tend to trust and engage more with others who belong to their perceived social group or network. Because Greg was an outsider to the recipients (compared to the staff sender), the emotional pull of his testimonial may have been weakened. Another contributing factor could be cognitive overload. The Treatment email included a personal story from Greg, which while emotionally powerful, might have added a layer of complexity or unfamiliarity that decreased its effectiveness compared to the more straightforward faux-forward email (the Control). The faux-forward email, which came from a familiar source, was simpler and less cognitively demanding for the reader, making it easier to process and act upon. This suggests that while donor testimonials are effective in fostering connection and trust, the familiarity of the sender is crucial in determining how much attention and engagement the email receives. If the recipient doesn't recognize or trust the sender immediately, they may be less likely to open the email and, as a result, less likely to engage with the content, including the donation link.

Experiment Documented by Ardee Coolidge

Question about experiment #195810

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.