Heritage Action for America

How emphasizing matching gifts during the serial ask impacted revenue and donor conversion.

Experiment ID: #12512

Heritage Action for America

Experiment Summary

Timeframe: 01/24/2019 - 02/28/2019

During the calendar year end campaign for Heritage, we installed an exit popup which used social proof language along with a symbolic gift ask to people looking to leave the page (on desktop computers only) to try to increase donor conversion rate and revenue increases. We wanted to see if we could increase revenue while maintaining the donor conversion rate when using language to reinforce matching your gift amount to Heritage Action to the same amount the donor just selected to give to the Heritage Foundation.

Research Question

There were actually two tethered research questions associated with this experiment, which were:

  • Will additional social proof language to reinforce matching your gift to HAFA increase revenue?
  • Will the higher gift ask amount positively or negatively impact the donor conversion rate?

Design

C: Control
T1: Treatment #1

Results

 Treatment NameRevenue per VisitorRelative DifferenceConfidenceAverage Gift
C: Control $1.81$54.66
T1: Treatment #1 $2.1920.8% 80.9%$86.33

This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was not met so these experiment results are not valid.

Key Learnings

Although the revenue lift did not fully validate we noticed two things, which were:

  1. With 80.8% level of confidence, we saw that the matching gift language increased revenue by 20.8%.
  2. With 100% level of confidence, we also saw that it decreased the instant donation conversion rate by -23.3%.

Although the revenue lift wasn’t validated, it’s enough to be directionally accurate. Our assumption was that this would decrease the donor conversion rate relatively to the lift in revenue, and it appears that our hypothesis was indeed true.

There are two additional noteworthy observations from this analysis, though:

In reviewing the control transactions, the symbolic gift amount of $17.76 represented 87 of the 449 transactions in the “control” segment of the experiment. If those gifts were removed, the control segment was about even with the number of transactions that were in the “treatment” segment (where the treatment would only have seen -3.62% difference).

Finally, when looking at the percentage of gifts that were matched between organizations across the two segments, only 70.8% of the gifts within the “control” segment were matched between the two organizations, but the “treatment” segment (which focused on emphasizing a matching gift between both organizations) boasted a 90.3% matching gift rate.

Relatively speaking, that would be an increase of +27.4% increase in matching gift rates for the treatment segment — which was the driving force between the increase in revenue for that segment.


Experiment Documented by Greg Colunga
Greg Colunga is Executive Vice President at NextAfter.

Question about experiment #12512

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.