How reducing the specificity of an offer affects clickthrough rate Experiment ID: #4937
Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.
Timeframe: 9/8/2016 - 9/12/2016
Hillsdale College was raising money in honor of Constitution Day, a holiday to mark the day the Constitution was signed. They had a new project they were excited about, The National Index of Constitutional Literacy. They created this initiative to measure the effectiveness of their popular online courses that teach Constitutional principles. While the project was specific, they wondered if it was too specific, and if a broader, “big vision” appeal might attract more interest from their file. They decided to create a treatment and let the data direct the general value proposition of their overall campaign.
They crafted a second appeal that included the Index of Constitutional Literacy, but as part of a big vision meant to reverse the marginalization of the Constitution in America. They split their file evenly and launched the two appeals, selecting clickthrough rate as the initial indicator of donor interest.
How will reducing the specificity of an offer (in favor of a big vision) affect clickthrough rate?
MECLABS Conversion Factors Targeted
C = 4m + 3v + 2( i - f) - 2a ©
Copyright 2015, MECLABS
|Treatment Name||Click Rate||Relative Difference||Confidence|
|C:||Project Specific Appeal||0.19%|
|T1:||Big Vision Appeal||1.7%||788.4%||100.0%|
This experiment has a required sample size of 319 in order to be valid. Since the experiment had a total sample size of 536,773, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
788.4% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
The treatment with the bigger vision, produced an significant increase in clickthrough rate—788.4%. This showed that Hillsdale donors, while interested in the project, showed much more interest in an initiative that was far-reaching in scope and much larger in impact. However, clickthrough rate is not the only metric at play here. If more people don’t give to the larger vision, then this experiment produces no actionable learning.
The team dug deeper into the data to find how reducing the specificity of the offer affected giving.