How reinforcing the value proposition for new subscribers through ad service during the welcome series increased downstream donor conversion rate
Americans for Prosperity
Experiment Summary
Ended On: 08/09/2019
We wanted to experiment with the value of reinforcing Americans for Prosperity’s value proposition for new subscribers as they participated in the New Subscriber Welcome Series to see if it would have any impact upon their program’s ability to convert new subscribers into first time donors.
Research Question
Would running brand value proposition ads over welcome series participants improve the new subscriber’s likelihood to convert into a first time donor for the organization?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Standard Welcome Series (no ads) | 0.14% | ||
| T1: | Reinforced Welcome Series (with value proposition ads for duration) | 1.2% | 759.9% | 99.5% |
This experiment has a required sample size of 142 in order to be valid. Since the experiment had a total sample size of 12,146, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 759.9% increase in conversion rate
× 0% increase in average gift
Key Learnings
With a 99.4% level of confidence, we observed an increase of +760% in downstream donor conversion when compared to the welcome series that did not also include brand value proposition ads running to welcome series participants.
Another very interesting observation we made with this experiment was how the treatment segment performed when measuring the time it took for the organization to secure the first gift from each audience.
The control segment (who received no ads during the welcome series) took an average 29.15 days to secure the first gift. Meanwhile, the treatment segment (who did receive ads during the welcome series) took an average of 17.55 days to secure the first gift.
This means that the treatment segment approach cut the time it took to convert a first time donor by 40%.
Finally, we also observed that the treatment segment’s average first gift was +11.5% higher than the control segment’s average first gift amount (note: the level of confidence on this portion of the experiment did not validate, but did achieve a 91.6% level of confidence, implying that it would have likely validated if the experiment continued).
To summarize: The value proposition reinforcement ads over welcome series messages being sent to new subscribers (not new donors):
- Increased the likelihood that someone would eventually convert into a first time donor,
- Increased the speed by which they made that decision, and also
- Increased the amount they decided to give for their first gift to the organization.
Some other things to note about the experience for the experiment is noted below:
- The incremental expense per person in the treatment experience increased costs by an average of about $1.50 in ads per participant included in the sample set.
- The campaign settings were on a fixed daily spend, rather than on a per participant level, which means that the average number of impressions per person in the treatment segment came out to 11.8 impressions per matched contact.
- There were no standard weekly (or monthly) direct solicitations sent to either audience for the duration of the experiment.
- There were no high urgency or seasonal fundraising campaigns that included direct solicitations sent to either audience for the duration of the experiment.
Final Recommendation(s):
- We would advise relaunching a second experiment that controls costs for the treatment group and looks to cut the cost/participant down by more than 50% of what we actually spent (where we would be targeting around $0.40 per participant in incremental ad services expenses) to bring the number of ad impressions down closer to about 5/participant for the duration of the experiment.
- We would advise exploring ways to generate a more even split in the segments.
Once these elements are implemented, we would advise that we re-run the experiment to see what this decreased investment does in terms of ad impression counts, as well as downstream performance related to downstream donor conversion, time to secure the first gift and average first gift metrics.
Further, we advise continuous monitoring of these newly converted donors for reevaluation periodically over the next 12-months to see what, if any, difference there is as it pertains to gift frequency, increase in giving amounts, and/or retention of these donors long-term, as well.
In light of these findings, however, it certainly seems advisable to install value proposition reinforcement ads over your new subscriber welcome series of emails, as it has a valid lift in downstream donor conversion rates and increases the speed in which the first time donor converts as opposed to the emails alone.