The Heritage Foundation

How the fear of loss of membership benefits increased donor conversion rate

Experiment ID: #10683

The Heritage Foundation

Experiment Summary

Timeframe: 02/08/2019 - 02/12/2019

Within the 2019 Heritage Foundation Kickoff Campaign, most of the control messaging delivered was highly transaction — largely focusing upon driving members online to renew their membership for the year. On the final day of the campaign, we wanted to test adjusting motivation for the file that had not yet given by treating the email with what they would lose if they didn’t renew their membership benefits, as opposed to the control message (which was tailoring motivation towards “hope of gain”).

Research Question

Will adjusting the motivation of a message to focus on the “fear of loss” increase donor conversion rates?

Design

C: Control

T1: Treatment #1

Results

	Treatment Name	Conv. Rate	Relative Difference	Confidence
C:	Control	0.64%
T1:	Treatment #1	0.78%	21.7%	99.3%

This experiment has a required sample size of 28,155 in order to be valid. Since the experiment had a total sample size of 106,066, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

0% increase in traffic
× 21.7% increase in conversion rate
× 0% increase in average gift

Key Learnings

With a 99.2% level of confidence, we successfully validated this experiment for the treatment ("fear of loss" message) with a 21.7% lift in donor conversion rates.

Other metrics impacted are noted as follows (all values shown are relative to the control version's performance in the same performance metric category):

Email Open Rate: +2.0% (with a 87.2% LoC)
Email Clickthrough Rate: -4.6% (with a 64.4% LoC)
On-Page Conversion Rate: +8.5%
Revenue/Visitor: +4.2%
Average Gift: +6.0%

Further experimentation should be attempted, however these results are encouraging that "fear of loss" may be a better motivator for members to renew their membership, as opposed to the "hope of gain."

We theorize that the personal impact of the loss is potentially felt more than the collective gain that we together can create. The tangibility of the impact of renewing their gift is more poignant, and is felt more -- which appears to have driven the increase in motivation to go online and renew.

These messages were sent on the morning of the final day. They both included referencing that it was the final day (e.g. created a sense of urgency to take action today), and structurally they were the same (in terms of sender, subject line, template, etc).

We would advise using a "fear of loss" message at least on the final day of a campaign, but can it provide good lift earlier in the campaign without weakening the final day message, as well? Further experimentation is recommended on timing within the campaign timeline.

Experiment Documented by Greg Colunga

Question about experiment #10683

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.