How video affects conversion on a campaign donation page
Hillsdale College
Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.
Experiment Summary
Timeframe: 06/24/2020 - 06/28/2020
Hillsdale College was running their fiscal year end campaign. Their team had generated a video designed to inspire giving, which they planned to use in cultivation leading up to the campaign. However, they wanted to test to see if it would improve conversion on the main donation page. Historically, they had seen that video decreased conversion on donation pages, but given the increase of their audience and the passage of time, they wanted to see if that still held true. They created an alternate donation page that included the video but held all other variables the same and split the traffic to determine the impact.
Research Question
How will video affect conversion on a campaign donation page?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | 36.1% | $0.00 | ||
| T1: | with Video | 28.6% | -20.9% | 98.8% | $0.00 |
This experiment has a required sample size of 307 in order to be valid. Since the experiment had a total sample size of 1,514, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 20.9% decrease in conversion rate
× 0% increase in average gift
Key Learnings
The treatment with the video reduced conversion by 20.9% in aggregate. Further segment analysis revealed that mobile traffic was the most affected, where the video reduced conversion by 37.5%.
There are many hypotheses as to why this happens. Often, the video might be a diversion from the real intent, which is giving. Or, on mobile, the video might overtake the screen entirely and remove the donation form and copy from view.
This shows that video, while there might be benefits in other ways to assist giving, does decrease conversion when added directly to the donation page.