How we evaluated the potential for email versus donor acquisition
Illinois Policy Institute
Experiment Summary
Timeframe: 03/17/2017 - 03/31/2017
In a previous experiment for Illinois Policy Institute, we had discovered the optimal tactic for email acquisition on their news website. By using a slide out newsletter offer, we were able to increase email acquisition by 470%.
However, email acquisition was one of several key performance indicators that the organization was looking to improve. Illinois Policy was also focused on finding ways to acquire donors from this same audience. With these two differing priorities, we decided to a launch a treatment that would focus on donor acquisition. The goal was to see how many email addresses would be given up by focusing on the donor acquisition element.
With this in mind, we launched an inline donation ask in the middle of various news articles. We called this the “Dear Reader” treatment and it talked about the nonprofit status of the organization and the other policy work that is being done in the state.
Research Question
Which offer (email vs. donor acquisition) will be more successful for the organization?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Email Slide Out | 0.00% | ||
| T1: | Dear Reader | 0.03% | 695.4% | 98.0% |
This experiment has a required sample size of 21,812 in order to be valid. Since the experiment had a total sample size of 61,378, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 695.4% increase in conversion rate
× 0% increase in average gift
Key Learnings
We discovered that the “Dear Reader” inline donation ask was able to boost the website’s donor conversion rate by 695%. There was always the ability to give a gift on the site but without the specific call out, visitors were not likely to take that action.
However, in prioritizing the donor acquisition over email acquisition, we discovered that IPI was giving up roughly 18 new email addresses for every donor that was acquired. When we compared this to the percentage of these new emails that eventually convert to a donor, we were then able to determine which offer would provide the most long-term value for the organization.