How radically redesigning a landing page with clarifying copy and removing navigation effects email acquisition rate
Luther Seminary
Experiment Summary
Timeframe: 06/16/2022 - 08/04/2022
Luther Seminary had been running their acquisition pages with navigation in the header and footer of the page. They wanted to test what would happen if we removed the navigation from the page to see if this would increase email acquisition.
We tested this concept on their Cosmic Adventure online course landing page. On top of this page having the navigation bars, the offer was not performing well for a consistent amount of time, so they also wanted to test changing the headline and body copy of the page to see if providing clarity for the user as to what they would receive in this course would help increase email acquisition.
The original page language was vague throughout the headline, sub-headline, and body copy. Within the new copy, we wanted to be very specific in the headline and subheadline by stating exactly what this course would do for them – give them answers to questions between science and faith through an informative course. Within the body copy, we wanted to focus on curiosity and clarity.
Research Question
We believe that removing the navigation and providing clarity in the page copy for users signing up for this online course will achieve a higher email acquisition rate.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | 37.0% | $0.00 | ||
| T1: | Same copy, no nav/footer | 40.8% | 10.3% | 73.6% | $0.00 |
| T2: | New copy, no nav/footer | 44.7% | 20.8% | 96.7% | $0.00 |
This experiment has a required sample size of 470 in order to be valid. Since the experiment had a total sample size of 1,140, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 20.8% increase in conversion rate
× 0% increase in average gift
Key Learnings
This experiment (treatment page 2) produced a valid increase in the email sign-up rate of 44.7% with a 96.6 level of confidence. In looking at all three variants, we believe that removing the navigation in the header and footer provided the biggest impact on our sign-up rate. On top of that, changing the page copy to be more clarifying for the user as to what this course will give them ultimately provided us with the valid increase needed.
It is recommended that this be tested on other acquisition pages separately, so we can see the true impact of removing navigation vs adding clarifying copy.
Notable changes to other metrics due to the experiment:
- Revenue experienced a 113.8% increase with a 86% level of confidence.