How aligning the value proposition with the audience affects clickthrough rate Experiment ID: #9445

Hillsdale College

Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.

Experiment Summary

Ended On: 9/19/2018

Hillsdale College was running a campaign to fund their new online course. They were trying to get traffic to the donation page from several of their properties that had inbound traffic. They used sticky bar banners to attempt to motivate visitors to give. They tested for different motivations: FirstThey wanted to ask users to help launch the course by asking a question.Then they tried a second treatment that was designed to provoke curiosity using “find out how “language. Third, they tried a command: “Help launch Hillsdale’s new Congress course”, and finally, they tried a more complex value proposition: “Help educate America for liberty with Hillsdale’s new course.” They launched the sticky bar banners as a four-way test to determine a winner on the Hillsdale Dialogues page, which attracts an audience of intellectually driven visitors who arrive to listen to a podcast.

Research Question

Which value proposition will deliver the most clicks?

Design

C: Control
T1: Treatment #1
T2: Treatment #2
T3: Treatment #3

Results

Treatment Name Click Rate Relative Difference Confidence
C: Control 1.7%
T1: Treatment #1 3.2% 90.5% 97.8%
T2: Treatment #2 2.4% 41.3% 75.4%
T3: Treatment #3 4.5% 169.5% 100.0%

This experiment has a required sample size of 538 in order to be valid. Since the experiment had a total sample size of 4,338, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    169.5% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The fourth value proposition, “Help educate America for liberty with Hillsdale’s new online course” delivered almost twice as many clicks as any of the other banners. This show that the edgy intellectually-driven audience responded well to a more complex value proposition than just a simple ask. It showed that they buy into the mission behind the course—not just the simple launch of a new course. Typically this would be an interesting, but not insightful learning, except for the fact that it was contradicted by another test on Hillsdale’s online courses page in which the “launch” language won. This shows the importance of testing across multiple properties with different messages so that you can match the right message to the right audience.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Jeff Giddens

Jeff is the President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.