How content offer illustrations impact email acquisition rate in a Facebook Ad Experiment ID: #6667

Hillsdale College

Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.

Experiment Summary

Timeframe: 5/16/2017 - 5/23/2017

Hillsdale College decided to take a new approach to producing one of their most popular online courses: Introduction to the Constitution. Instead of showing the teacher lecturing from a podium in a classroom setting, they presented the teacher—in this case, Dr. Larry Arnn—sitting at a table discussing the material with his students. This groundbreaking new style immerses the viewer in the Hillsdale classroom experience.

Since this was a completely different approach than before, they were interested in learning how the general public would respond to the change. To discover whether this “insider” view of the classroom would resonate with their audience, they launched a Facebook A/B test to determine what type of imagery would increase course signups.

Research Question

Will the “Introduction to the Constitution” course illustration attract more people than an actual image of the course setting?


C: Control
T1: Treatment 1


Treatment Name Conv. Rate Relative Difference Confidence
C: Control 0.11%
T1: Treatment 1 0.20% 85.5% 100.0%

This experiment has a required sample size of 16,459 in order to be valid. Since the experiment had a total sample size of 346,410, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 85.5% increase in conversion rate
× 0% increase in average gift

Key Learnings

Hillsdale found that the course illustration produced an 85.5% increase in email acquisition, and thus, course signups. Though the new approach to producing this course has resonated well overall with Hillsdale College’s target audience, it’s apparent that showing a “quick glance” of the course in action via a Facebook ad was not well received. It’s clear that the course illustration produced a higher level of intrigue, amounting in not only a higher clickthrough rate but also a higher email acquisition rate.

Hillsdale’s audience may have shied away from the image of the course because it was radically different than previous course advertisements. The “quick glance” may have, in fact, caused more confusion about the value proposition and added to the anxiety of the viewer. These results have prompted further testing across other courses to determine the importance of course illustrations in Facebook advertising.

Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.

Experiment Documented by...

Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.