How updating the copy and design of a landing page effects email conversion rate | NextAfter
Luther Seminary

How updating the copy and design of a landing page effects email conversion rate

Experiment ID: #64014

Luther Seminary

Experiment Summary

Timeframe: 06/14/2021 - 07/04/2021

With this experiment, we are testing Luther Seminary’s daily devotional called GodPause. This was a radical redesign where we tested removing the header tabs, footer tabs, added the same image we used on the Facebook ad, shortened the form field length, as well as updated the copy on the page. With the copy changes, we did use the word GodPause as new names don’t know what GodPause is. So we changed the copy to talk more about the use case and benefits of this daily devotional.

Research Question

We believe that a radical redesign for the sign up page will achieve an increase in email conversion rate.

Design

C: Control
T1: Radical Redesign - New Copy

Results

  Treatment Name Conv. Rate Relative Difference Confidence Average Gift
C: Control 40.9% $0.00
T1: Radical Redesign - New Copy 48.1% 17.7% 99.4% $0.00

This experiment has a required sample size of 362 in order to be valid. Since the experiment had a total sample size of 1,409, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 17.7% increase in conversion rate
× 0% increase in average gift

Key Learnings

Since this was a radical redesign we are unable to point to a specific reason for the 17% increase in conversion rate. My hypothesis is that it is a combination of the updated copy as well as removing the header tabs.

Notable changes to other metrics due to the experiment:

  • Donations experienced a 57.8% decrease with a 91% level of confidence.

Observations of the impact on other visitor segments in the experiment:

  • (Other) visitors had a 48.5% increase in email acquisitions with a 98% level of confidence.
  • Desktop visitors had a 29.2% increase in email acquisitions with a 100% level of confidence.
  • New visitors had a 21.4% increase in email acquisitions with a 100% level of confidence.
  • Organic Search visitors had a 33.3% increase in email acquisitions with a 92% level of confidence.
  • 45-54 visitors had a 40.5% decrease in email acquisitions with a 92% level of confidence.
  • Direct visitors had a 100.0% increase in email acquisitions with a 100% level of confidence.


Experiment Documented by NextAfter

Question about experiment #64014

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.