How a question in a headline affects conversion rate on an acquisition page Experiment ID: #9003

National Breast Cancer Foundation

The National Breast Cancer Foundation's mission is to provide help and inspire hope to those affected by breast cancer through early detection, education, and support services.

Experiment Summary

Timeframe: 3/20/2018 - 4/19/2018

NBCF was offering free healthy living tips as an acquisition offer, and wanted to test whether adding a question in the headline would increase people’s motivation to download the offer.

Research Question

Does asking a question in a headline convert more people than an actionable value prop?


C: control
T1: question headline


Treatment Name Conv. Rate Relative Difference Confidence
C: control 49.7%
T1: question headline 40.2% -19.1% 100.0%

This experiment has a required sample size of 212 in order to be valid. Since the experiment had a total sample size of 2,049, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 19.1% decrease in conversion rate
× 0% increase in average gift

Key Learnings

The question headline decreased conversion by 19%. It’s possible that because we were asking a question rather than giving them an actionable charge, “Start living healthier this week,” that people were less motivated by the headline to convert.

Experiment Documented by...

Allison Jones

Allison is an Optimization Associate at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.