A Public Media Organization

Does a pop-up with future framing copy generate more donations than a contrast effect copy?

Experiment ID: #195832

A Public Media Organization

Experiment Summary

Timeframe: 03/10/2025 - 04/01/2025

This experiment was conducted for WUNC, a public radio station serving North Carolina, to evaluate the impact of homepage pop-up messaging on donor acquisition. The goal was to determine whether leaning into urgency and future challenges would outperform messaging focused on WUNC’s lack of a paywall.

Research Question

We believed that emphasizing urgency and the challenges ahead would better motivate donations than the paywall-focused message, as it could create a stronger emotional incentive and sense of immediacy for users.

Psychological Framing

This test compared two psychological tactics:

Future Framing (Test): Highlights what lies ahead—upcoming threats, opportunities, or needs. This approach encourages users to act now to influence future outcomes.
Contrast Effect (Control): Draws attention to what users currently have (free access) and implies a risk of losing it. This technique relies on juxtaposing the current benefit with the potential consequences of inaction.

Design

C: Control

T1: Treatment 1

Results

	Treatment Name	Conv. Rate	Relative Difference	Confidence	Average Gift
C:	Control	0.76%			$44.70
T1:	Treatment 1	1.0%	32.7%	87.4%	$64.31

This experiment has a required sample size of 10,845 in order to be valid. Since the experiment had a total sample size of 13,176, and the level of confidence is not above 95% the experiment results are not valid.

Key Learnings

Main Takeaway

The urgency-focused copy led to stronger performance than the paywall-based control. The experiment suggests a strong directional insight favoring the test copy. The test also outperformed the control in key revenue and donor quality metrics, including sustainers and new donors.

Performance Metrics and Insights:

Total Conversions: Includes those who clicked main donate after seeing a version +33% lift with 87% confidence Donation Breakdown:

Total Direct Gifts: +20% with a 88% confidence
One-Time Gifts: –28.6% with a 43.6% confidence
One-Time Revenue: +41.2%
Sustainers: +133.3% with a 79% confidence
Monthly Revenue (Annualized): +177.8%
New Donors: +300% with an 82% confidence

Note on Statistical Reliability: While the revenue metrics show very high confidence levels, the low sample sizes (5–7 gifts per group) and low variation between values reduce the reliability of these specific results. These findings should be treated as directional insights, not conclusive outcomes. Additional testing with larger samples is recommended.

Observations:

While total conversions did not achieve full statistical significance, the test consistently outperformed the control across most donor quality metrics.
The urgency-based message likely connected more emotionally with users, prompting not only more gifts, but also stronger long-term support through sustainer commitments.

Next Steps

Roll Out Winning Copy: Implement urgency-focused messaging across homepage and other key entry points.
Iterate on Messaging: Refine urgency language and test different variations (e.g., future threats, upcoming funding needs).
Test Other Formats: Use the test language in different pop-up designs, placements, or even embedded CTAs to evaluate scalability.

This experiment highlights the power of urgency-based messaging to drive more conversions, higher-quality gifts, and greater acquisition—all crucial during fundraising campaigns and periods of public uncertainty.

Experiment Documented by NextAfter

Question about experiment #195832

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.