How reducing the specificity of an offer affects clickthrough rate Experiment ID: #4937

Hillsdale College

Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.

Experiment Summary

Timeframe: 9/8/2016 - 9/12/2016

Hillsdale College was raising money in honor of Constitution Day, a holiday to mark the day the Constitution was signed. They had a new project they were excited about, The National Index of Constitutional Literacy. They created this initiative to measure the effectiveness of their popular online courses that teach Constitutional principles. While the project was specific, they wondered if it was too specific, and if a broader, “big vision” appeal might attract more interest from their file. They decided to create a treatment and let the data direct the general value proposition of their overall campaign.

They crafted a second appeal that included the Index of Constitutional Literacy, but as part of a big vision meant to reverse the marginalization of the Constitution in America. They split their file evenly and launched the two appeals, selecting clickthrough rate as the initial indicator of donor interest.

Research Question

How will reducing the specificity of an offer (in favor of a big vision) affect clickthrough rate?

MECLABS Conversion Factors Targeted

C = 4m + 3v + 2( i - f) - 2a ©

Copyright 2015, MECLABS

Design

C: Project Specific Appeal
T1: Big Vision Appeal

Results

Treatment Name Click Rate Relative Difference Confidence
C: Project Specific Appeal 0.19%
T1: Big Vision Appeal 1.7% 788.4% 100.0%

This experiment has a required sample size of 319 in order to be valid. Since the experiment had a total sample size of 536,773, and the level of confidence is above 95%, the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    788.4% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The treatment with the bigger vision, produced an significant increase in clickthrough rate—788.4%. This showed that Hillsdale donors, while interested in the project, showed much more interest in an initiative that was far-reaching in scope and much larger in impact. However, clickthrough rate is not the only metric at play here. If more people don’t give to the larger vision, then this experiment produces no actionable learning.

The team dug deeper into the data to find how reducing the specificity of the offer affected giving.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Jeff Giddens

Jeff is a Senior Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.