Illinois Policy Institute

How reducing the number of calls to action impacted clicks in a cultivation message

Experiment ID: #124980

Illinois Policy Institute

Experiment Summary

Ended On: 02/01/2023

When preparing a cultivation message around a specific topic of interest, we found the organization offered a lot of relevant resources. The control copy gave the reader a full menu to select from. They could listen to a podcast, read a post, or explore stories from people personally impacted by the program. We wondered if having this many choices could result in decision fatigue and cause the reader to not engage with ANY of the options. We developed a treatment to test this theory and isolated the choice to read a single article. We believed the treatment would help the reader have clarity by establishing a clear call to action. We split the audience 50/50 and measured the impact on clicks.

Research Question

We believe that presenting a single call to action for cultivation message readers will achieve an increase in engagement because the isolated option will reduce potential friction caused by decision fatigue.


C: Control
T1: Treatment #1


 Treatment NameClick RateRelative DifferenceConfidence
C: Control 0.21%
T1: Treatment #1 0.41%95.8% 100.0%

This experiment has a required sample size of 6,011 in order to be valid. Since the experiment had a total sample size of 329,829, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    95.8% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

What we discovered through this experiment was the treatment resulted in a 95.8% increase in clicks with a 100% statistical level of confidence. The key learning from this experiment is that presenting a single call to action for cultivation message readers can significantly increase engagement by reducing decision fatigue. This means that the control copy that offered a full menu of options may have overwhelmed readers and resulted in lost engagement opportunities. In future experimentation, it is recommended to simplify the call to action and reduce options to reduce the risk of decision fatigue and improve engagement rates.

Experiment Documented by Rebekah Josefy
Rebekah Josefy is an Optimization Director at NextAfter.

Question about experiment #124980

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.