How a softer call to action affects clickthrough rate
Hillsdale College
Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.
Experiment Summary
Timeframe: 11/12/2015 - 11/16/2015
Hillsdale College had launched a new online course about the life and writings of acclaimed author and apologist C.S. Lewis. They wanted to promote these on Facebook, as they had done very successfully with other courses. But they wanted to better understand which calls to action drove the greatest amount of traffic to the page. Previously, they had used “activation” language like “Start Your Free Course Now” or “Get Instant Access Now”. They decided to test whether their audiences were actually motivated at that point in the funnel to “activate” — or if a softer call to action like “Learn More” would drive more traffic and conversions. So they set up two ads and set out to gather enough data to determine a result.
Research Question
Will a softer call to action like “Learn More” drive more traffic than an “activation” ask like “Get Free Access Now”?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 5.2% | ||
| T1: | Treatment 1 | 6.7% | 29.7% | 100.0% |
This experiment has a required sample size of 1,794 in order to be valid. Since the experiment had a total sample size of 21,512, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
29.7% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The results tell us that at the first step in the journey, some of our audience is “sold” enough to be able to take that first step with an “activation” message. However, that leaves a lot of prospective users on the table who might want to learn more about the course before committing to sign up for it.