How a new serial ask primary message decreased revenue Experiment ID: #15016

Heritage Action for America

Experiment Summary

Ended On: 4/23/2019

Heritage Action is committed to continuously experimenting with the primary serial ask message, tactics and techniques to improve serial ask donor conversion and revenue opportunities from The Heritage Foundation donors. Our latest iteration was focused on creating an emotional appeal to the Heritage donor viewing the serial ask to determine whether or not we could test our way into a new message to improve results.

We wanted to focus on an emotional appeal, related to something centered around the organization’s key value proposition: Accountability.

Research Question

Would an emotional appeal about the need for increased accountability for lawmakers because Washington is corrupt improve serial ask donor conversion rates or revenue?

Design

C: Control
T1: Treatment #1

Results

Treatment Name Revenue per Visitor Relative Difference Confidence Average Gift
C: Control $0.19 $43.86
T1: Treatment #1 $0.10 -44.9% 95.5% $44.55

This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was met so these experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 45.7% decrease in conversion rate
× 1.6% increase in average gift
44.9% decrease in revenue

Key Learnings

With a 95.4% level of confidence, we saw a decrease in serial ask revenue through the treatment message by -44.9%.

Although the average gift increased slightly (+1.6%), the decrease in donor conversion rate (-50.3%, with a 94.7% level of confidence), the treatment brought the overall revenue down for the treatment.

It should be noted that for the last 4-days of this experiment, The Heritage Foundation did introduce a new recurring gift popup experience prior to the serial ask page, which may have negatively impacted these results. That said, the preliminary trend prior to this experience being introduced followed a similar trend line, so for that reason, we decided to end the experiment with a validated revenue decrease.

Continued experimentation with primary serial ask messaging is advised.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Greg Colunga

Greg is Executive Vice President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.