How varying types of third party credibility affects email acquisition Experiment ID: #6441
Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.
Timeframe: 3/6/2017 - 4/6/2017
Hillsdale launched a course designed to give students a better understanding of the role of the Executive Branch. Previous optimization efforts from a similar test on a different course had already shown that student validation outperformed the presence of a course schedule by 20.5%. They were interested in discovering how student quotes as third-party validation would perform against an endorsement from former President Ronald Reagan. They hypothesized that student quotes would outperform an endorsement from a former president because it was more personal, experienced-based, and spoke from the point of view of the page visitor.
They launched an A/B test to find out which treatment produced the highest email acquisition rate.
Will student quotes outperform a endorsement from a former president?
MECLABS Conversion Factors Targeted
C = 4m + 3v + 2( i - f) - 2a ©
Copyright 2015, MECLABS
|Treatment Name||Conv. Rate||Relative Difference||Confidence|
This experiment has a required sample size of 20,807 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
The results of this test were statistically insignificant. Even though the treatment with student quotes increased conversion by a 2%, the test could not be validated, which showed Hillsdale that prospects were not heavily influenced by varying types of third party validation. This prompted a reassessment of which page elements should be further tested.