How content offer illustrations impact email acquisition rate in a Facebook Ad Experiment ID: #6667
Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.
Timeframe: 5/16/2017 - 5/23/2017
Hillsdale College decided to take a new approach to producing one of their most popular online courses: Introduction to the Constitution. Instead of showing the teacher lecturing from a podium in a classroom setting, they presented the teacher—in this case, Dr. Larry Arnn—sitting at a table discussing the material with his students. This groundbreaking new style immerses the viewer in the Hillsdale classroom experience.
Since this was a completely different approach than before, they were interested in learning how the general public would respond to the change. To discover whether this “insider” view of the classroom would resonate with their audience, they launched a Facebook A/B test to determine what type of imagery would increase course signups.
Will the “Introduction to the Constitution” course illustration attract more people than an actual image of the course setting?
|Treatment Name||Conv. Rate||Relative Difference||Confidence|
This experiment has a required sample size of 16,459 in order to be valid. Since the experiment had a total sample size of 346,410, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 85.5% increase in conversion rate
× 0% increase in average gift
Hillsdale found that the course illustration produced an 85.5% increase in email acquisition, and thus, course signups. Though the new approach to producing this course has resonated well overall with Hillsdale College’s target audience, it’s apparent that showing a “quick glance” of the course in action via a Facebook ad was not well received. It’s clear that the course illustration produced a higher level of intrigue, amounting in not only a higher clickthrough rate but also a higher email acquisition rate.
Hillsdale’s audience may have shied away from the image of the course because it was radically different than previous course advertisements. The “quick glance” may have, in fact, caused more confusion about the value proposition and added to the anxiety of the viewer. These results have prompted further testing across other courses to determine the importance of course illustrations in Facebook advertising.