First Baptist Dallas

How using value prop language in an email impacted click-through and donation conversion rates for Pathway to Victory

Experiment ID: #93744

First Baptist Dallas

Experiment Summary

Timeframe: 05/10/2022 - 05/24/2022

Pathway to Victory is a ministry who has built a fundraising program around the offer of premiums—books, pamphlets, brochures, etc. Most of the language used in email appeals has traditionally been focused on the premium offer itself, with mentions of the book, what it contains, and different ways to get the book. For this experiment we wanted to test whether making the email more value-prop focused, with a reference to the book, would impact click-through and donor conversion rates.

This test was run with multiple emails; the results in this experiment are for the 1st email send, including a resend. Both emails pointed to the exact same landing page, which focused on the premium offer.

Research Question

We believe that adding value prop language for an email audience will achieve a higher click-through rate.

Design

C: Control Premium Focused
T1: Value Prop

Results

  Treatment Name Click Rate Relative Difference Confidence
C: Control Premium Focused 1.1%
T1: Value Prop 2.2% 106.2% 100.0%

This experiment has a required sample size of 945 in order to be valid. Since the experiment had a total sample size of 98,253, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    106.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The results of this experiment were extremely interesting. While there was a 106% increase in click-through rates total, with a 100% level of confidence, when looking at the experiment through a donation conversion lens, the results were different.

When looking at open-to-donation conversion rates, there was a 29% lift in donation conversion rates with the value prop version, with a 70% level of confidence. However, when looking at how many people clicked a link and actually completed the donation, there was a 37% DECREASE in donation conversions, with a 92% level of confidence.

Our hypothesis is that this result is that while the value prop version was driving more clicks to the landing page, without having any value prop language on the page itself, those potential donors lost their interest in making a donation and receiving the premium.

This then led to the next round of testing, which incorporated value prop language onto the landing page itself.


Experiment Documented by NextAfter

Question about experiment #93744

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.