NextAfter

How streamlined copy affects conversion on a benchmark survey page

Experiment ID: #10619

NextAfter

Experiment Summary

Timeframe: 01/21/2019 - 02/04/2019

In this experiment, we were trying to get more people to share their 3 key online fundraising metrics for us to create a new online fundraising benchmark report. Our original page had an in-depth description of the 3 key metrics, their role in the fundraising process, and a breakdown of how they each can affect revenue.

We wondered if we could streamline the copy by removing this section explaining the metrics in detail. The hypothesis was that it made the page feel more like a blog, and might be distracting from the final call-to-action.

Research Question

Will streamlining the landing page copy lead to greater enrollments in benchmark report?

Design

C: Description of Key Metrics
T1: No Description of Key Metrics

Results

 Treatment NameConv. RateRelative DifferenceConfidence
C: Description of Key Metrics 4.3%
T1: No Description of Key Metrics 9.1%110.0% 96.8%

This experiment has a required sample size of 217 in order to be valid. Since the experiment had a total sample size of 495, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 110.0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The treatment saw a 110% increase in people enrolling in the benchmark study. This will lead to significant increases in enrollment when we do a much more direct marketing push later on.

It also may be an indication that there is such thing as too much information. Explaining the full details of why we chose these 3 key metrics could be causing people to forget about the purpose of the page, what brought them there, and ultimately cause them to disengage. The additional value of this explanation is out-weighed by the additional friction it creates.


Experiment Documented by Nathan Hill
Nathan Hill is Vice President, NextAfter Institute.

Question about experiment #10619

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.