How removing visual breaks in a page did not increase email signup rate on a pledge
Americans for Prosperity
Experiment Summary
Timeframe: 10/24/2019 - 11/08/2019
As a part of experimenting with what will increase email signups from the Freedom Pledge offer for Americans for Prosperity, we theorized that the visual breaks in the design may have been creating “mental hitches” for the reader that was limiting our ability to acquire emails.
Research Question
Would removing the design elements and visual breaks on a pledge page increase email signup rate?
Design
Results
Treatment Name | Conv. Rate | Relative Difference | Confidence | |
---|---|---|---|---|
C: | Control | 55.1% | ||
T1: | Clean Treatment | 54.6% | -0.9% | 6.8% |
This experiment has a required sample size of 86,018 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
After running this experiment for more than 2 weeks, there was not a significant enough difference between the control and treatment experience to validate this hypothesis, showing that there was a difference of less than 1%, with a level of confidence of only 6.75%, as well.
Further experimentation may be required.
Question about experiment #18851
If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.