Americans for Prosperity

How removing visual breaks in a page did not increase email signup rate on a pledge

Experiment ID: #21354

Americans for Prosperity

Experiment Summary

Timeframe: 10/24/2019 - 11/8/2019

As a part of experimenting with what will increase email signups from the Freedom Pledge offer for Americans for Prosperity, we theorized that the visual breaks in the design may have been creating “mental hitches” for the reader that was limiting our ability to acquire emails.

Research Question

Would removing the design elements and visual breaks on a pledge page increase email signup rate?

Design

C: Control
T1: Clean Treatment

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: 55.1%
T1: Clean Treatment 54.6% -0.9% 6.8%

This experiment has a required sample size of 86,018 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

After running this experiment for more than 2 weeks, there was not a significant enough difference between the control and treatment experience to validate this hypothesis, showing that there was a difference of less than 1%, with a level of confidence of only 6.75%, as well.

Further experimentation may be required.


Experiment Documented by Greg Colunga
Greg Colunga is Executive Vice President at NextAfter.

Question about experiment #6421

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.