How removing styling from email affects donor conversion rate Experiment ID: #4965
Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.
Timeframe: 9/2/2016 - 9/15/2016
Hillsdale College was fundraising during their annual Constitution Day campaign. In previous emails, they had found that stripping out branding from their emails improved clickthrough rate and donor conversion rate. Their email template had been stripped to the bare essentials, yet still had a border around it with a separate background color. They decided to test this against a pure plain-text email that removed all styling entirely.
They split their file evenly and sent an A/B test to determine a winner.
Will removing styling from email positively affect donor conversion rate?
|Treatment Name||Conv. Rate||Relative Difference||Confidence|
This experiment has a required sample size of 320,212 in order to be valid. Since the experiment had a total sample size of 498,494, and the level of confidence is not above 95% the experiment results are not valid.
The treatment, with all styling removed, produced a 19% decrease in donor conversion rate. Though this experiment wasn’t completely valid, it was just on the cusp of being valid. Hillsdale hypothesized that the decrease in conversion rate was due to the way that different email clients render a plain-text email—there is no text wrapping and the email becomes harder to read. While the control template might indicate to some that it is a marketing email, the borders make the text more readable and therefore improve response.