How streamlined copy affects conversion on a benchmark survey page Experiment ID: #11650
Timeframe: 1/21/2019 - 2/4/2019
In this experiment, we were trying to get more people to share their 3 key online fundraising metrics for us to create a new online fundraising benchmark report. Our original page had an in-depth description of the 3 key metrics, their role in the fundraising process, and a breakdown of how they each can affect revenue.
We wondered if we could streamline the copy by removing this section explaining the metrics in detail. The hypothesis was that it made the page feel more like a blog, and might be distracting from the final call-to-action.
Will streamlining the landing page copy lead to greater enrollments in benchmark report?
|Treatment Name||Open Rate||Relative Difference||Confidence|
|C:||Description of Key Metrics||4.3%|
|T1:||No Description of Key Metrics||9.1%||110.0%||96.8%|
This experiment has a required sample size of 217 in order to be valid. Since the experiment had a total sample size of 495, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
110.0% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
The treatment saw a 110% increase in people enrolling in the benchmark study. This will lead to significant increases in enrollment when we do a much more direct marketing push later on.
It also may be an indication that there is such thing as too much information. Explaining the full details of why we chose these 3 key metrics could be causing people to forget about the purpose of the page, what brought them there, and ultimately cause them to disengage. The additional value of this explanation is out-weighed by the additional friction it creates.