Americans for Prosperity

How shortening the length of content impacted email acquisition

Experiment ID: #6502

Americans for Prosperity

Experiment Summary

Timeframe: 07/18/2018 - 07/25/2018

The Open Letter to the Senate had been a high performing email acquisition campaign for Americans for Prosperity for several weeks now. With an email acquisition rate higher than 60%, we had found it difficult to improve results through optimization.

Based upon the high motivation of the visitors coming to the page, we decided to see if simplifying the page copy could improve the results. The hypothesis was that, given the high motivation of visitors, they did not need as much value proposition to make the conversion. Our goal was to maintain the value expressed through the copy but shorten it up so that new visitors would find it easier to convert.

Research Question

Will shortening the value proposition on the page increase email acquisition rates?

Design

C: open letter
T1: shortened version

Results

 Treatment NameConv. RateRelative DifferenceConfidence
C: open letter 62.1%
T1: shortened version 63.3%1.9% 95.4%

This experiment has a required sample size of 12,285 in order to be valid. Since the experiment had a total sample size of 25,474, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 1.9% increase in conversion rate
× 0% increase in average gift

Key Learnings

It would appear that the length of the page did impact the likelihood of a highly motivated audience to convert, albeit only a minimal amount. We were able to statistically prove that the reduced length of copy increased conversion rate by 1.9%.

This is a unique situation where the motivation of the visitors was so high, they did not need much convincing. When audiences are less motivated, when the conversion rate is not as high, increasing the perceived value of the offer (even if it makes the page longer) has proven to be a more effective method for boosting conversion than getting the form “above the fold.” The lift of only 1.9% is evidence of just how minimally impactful the friction associated with copy length is to the conversion process.


Experiment Documented by NextAfter

Question about experiment #6502

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.