How updating the copy and design of a landing page effects email conversion rate
Luther Seminary
Experiment Summary
Timeframe: 06/14/2021 - 07/04/2021
With this experiment, we are testing Luther Seminary’s daily devotional called GodPause. This was a radical redesign where we tested removing the header tabs, footer tabs, added the same image we used on the Facebook ad, shortened the form field length, as well as updated the copy on the page. With the copy changes, we did use the word GodPause as new names don’t know what GodPause is. So we changed the copy to talk more about the use case and benefits of this daily devotional.
Research Question
We believe that a radical redesign for the sign up page will achieve an increase in email conversion rate.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | 40.9% | $0.00 | ||
| T1: | Radical Redesign - New Copy | 48.1% | 17.7% | 99.4% | $0.00 |
This experiment has a required sample size of 362 in order to be valid. Since the experiment had a total sample size of 1,409, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 17.7% increase in conversion rate
× 0% increase in average gift
Key Learnings
Since this was a radical redesign we are unable to point to a specific reason for the 17% increase in conversion rate. My hypothesis is that it is a combination of the updated copy as well as removing the header tabs.
[ADD THE FULL EXPERIMENT ANALYSIS HERE]
Notable changes to other metrics due to the experiment:
- Donations experienced a 57.8% decrease with a 91% level of confidence.
Observations of the impact on other visitor segments in the experiment:
- (Other) visitors had a 48.5% increase in email acquisitions with a 98% level of confidence.
- Desktop visitors had a 29.2% increase in email acquisitions with a 100% level of confidence.
- New visitors had a 21.4% increase in email acquisitions with a 100% level of confidence.
- Organic Search visitors had a 33.3% increase in email acquisitions with a 92% level of confidence.
- 45-54 visitors had a 40.5% decrease in email acquisitions with a 92% level of confidence.
- Direct visitors had a 100.0% increase in email acquisitions with a 100% level of confidence.