How a high urgency ask on a pop up affects click rate Experiment ID: #7972

National Breast Cancer Foundation

The National Breast Cancer Foundation's mission is to provide help and inspire hope to those affected by breast cancer through early detection, education, and support services.

Experiment Summary

Timeframe: 10/26/2017 - 10/31/2017

National Breast Cancer Foundation was in the final days of their BCAM campaign, and wanted to run a test using high urgency copy and a countdown clock on the popup that was running on all pages.

Research Question

Does increasing urgency in copy and with a countdown clock increase click rate to the donation page?

Design

C: Not high urgency
T1: High urgency with countdown clock

Results

Treatment Name Click Rate Relative Difference Confidence
C: Not high urgency 6.2%
T1: High urgency with countdown clock 1.8% -70.8% 100.0%

This experiment has a required sample size of 165 in order to be valid. Since the experiment had a total sample size of 73,630, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    70.8% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The high urgency treatment with the countdown clock decreased clicks by 70%. Since the design on the high urgency ad was also different than the control, it’s hard to be sure which specific part of the ad deterred people from clicking, but it’s likely that the new design paired with the very direct ask led only the very most donor-motivated people to click through.


Experiment Documented by...

Allison Jones

Allison is an Optimization Associate at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.