Illinois Policy Institute

How we evaluated the potential for email versus donor acquisition

Experiment ID: #2729

Illinois Policy Institute

Experiment Summary

Timeframe: 03/17/2017 - 03/31/2017

We discovered that the “Dear Reader” inline donation ask was able to boost the website’s donor conversion rate by 695%. There was always the ability to give a gift on the site but without the specific call out, visitors were not likely to take that action.

However, in prioritizing the donor acquisition over email acquisition, we discovered that IPI was giving up roughly 18 new email addresses for every donor that was acquired. When we compared this to the percentage of these new emails that eventually convert to a donor, we were then able to determine which offer would provide the most long-term value for the organization.

Research Question

Which offer (email vs. donor acquisition) will be more successful for the organization?


C: Email Slide Out
T1: Dear Reader


  Treatment Name Conv. Rate Relative Difference Confidence
C: Email Slide Out 0.00%
T1: Dear Reader 0.03% 695.4% 98.0%

This experiment has a required sample size of 21,812 in order to be valid. Since the experiment had a total sample size of 61,378, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 695.4% increase in conversion rate
× 0% increase in average gift

Key Learnings

In a previous experiment for Illinois Policy Institute, we had discovered the optimal tactic for email acquisition on their news website. By using a slide out newsletter offer, we were able to increase email acquisition by 470%.

However, email acquisition was one of several key performance indicators that the organization was looking to improve. Illinois Policy was also focused on finding ways to acquire donors from this same audience. With these two differing priorities, we decided to a launch a treatment that would focus on donor acquisition. The goal was to see how many email addresses would be given up by focusing on the donor acquisition element.

With this in mind, we launched an inline donation ask in the middle of various news articles. We called this the “Dear Reader” treatment and it talked about the nonprofit status of the organization and the other policy work that is being done in the state.

Experiment Documented by NextAfter

Question about experiment #2729

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.