How a pledge at the end of an article impacts recurring donor conversion rate
Leadership Institute
Experiment Summary
Ended On: 08/21/2020
The Leadership Institute runs a daily news website called Campus Reform, where leaders can read articles that covers what is happening on college campuses related to free speech and education.
With two subscription types (daily and weekly), there are a substantial number of visitors to this website each year.
Readers have long been presented with an “end of article” ad that asks them to become a supporter of the free journalism that they have just received. The donation page after the ad click is asking supporters to activate a monthly gift.
Could we increase the number of monthly donors we secured by putting an intermediary step in the process, while also building motivation through a pledge to propel them towards a higher donor conversion rate.
Research Question
Would a pledge at the end of an article designed to increase donor motivation increase recurring donor conversion rate?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | End of Article - Direct Ask | 0.00% | ||
| T1: | End of Article - Pledge | 0.00% | 8,687.3% | 98.5% |
This experiment has a required sample size of 8,471 in order to be valid. Since the experiment had a total sample size of 4,449,540, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 8,687.3% increase in conversion rate
× 0% increase in average gift
Key Learnings
This treatment experience ran for just 10 days and increased the recurring donor conversion rate by +8,687.3% with a 98.4% level of confidence.
The data table above actually shows the control’s performance between 1/1/2020 through 8/11/2020 (when the experiment was launched). Subsequently, during the experiment, we did not receive a recurring gift through the control experience. Therefore, we decided to open up the data collected in this study to review the performance for the control variant year-to-date.
So, simply put:
- The control experience (which has been running all year) produced one recurring gift for all of this year to the point that we launched the treatment experience.
- And after launching the treatment to just half of the web visitors, we were able to achieve 6 recurring donors in just 10 days.
This is a massive lift in results achieved in a very short period of time, in spite of the fact that only half of the website’s visitors are allowed to experience the treatment ad type.
Our recommendation is to roll this out to the full audience and continue iteratively testing to increase the number of people that decide to sign the pledge (starting the recurring donor acquisition funnel) and replace the control experience with this approach right away.