How using value prop language in an email impacted click-through and donation conversion rates for Pathway to Victory
First Baptist Dallas
Experiment Summary
Timeframe: 05/10/2022 - 05/24/2022
Pathway to Victory is a ministry who has built a fundraising program around the offer of premiums—books, pamphlets, brochures, etc. Most of the language used in email appeals has traditionally been focused on the premium offer itself, with mentions of the book, what it contains, and different ways to get the book. For this experiment we wanted to test whether making the email more value-prop focused, with a reference to the book, would impact click-through and donor conversion rates.
This test was run with multiple emails; the results in this experiment are for the 1st email send, including a resend. Both emails pointed to the exact same landing page, which focused on the premium offer.
Research Question
We believe that adding value prop language for an email audience will achieve a higher click-through rate.
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control Premium Focused | 1.1% | ||
| T1: | Value Prop | 2.2% | 106.2% | 100.0% |
This experiment has a required sample size of 945 in order to be valid. Since the experiment had a total sample size of 98,253, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
106.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The results of this experiment were extremely interesting. While there was a 106% increase in click-through rates total, with a 100% level of confidence, when looking at the experiment through a donation conversion lens, the results were different.
When looking at open-to-donation conversion rates, there was a 29% lift in donation conversion rates with the value prop version, with a 70% level of confidence. However, when looking at how many people clicked a link and actually completed the donation, there was a 37% DECREASE in donation conversions, with a 92% level of confidence.
Our hypothesis is that this result is that while the value prop version was driving more clicks to the landing page, without having any value prop language on the page itself, those potential donors lost their interest in making a donation and receiving the premium.
This then led to the next round of testing, which incorporated value prop language onto the landing page itself.