How reducing the number of calls to action impacted clicks in a cultivation message
Illinois Policy Institute
Experiment Summary
Ended On: 02/01/2023
When preparing a cultivation message around a specific topic of interest, we found the organization offered a lot of relevant resources. The control copy gave the reader a full menu to select from. They could listen to a podcast, read a post, or explore stories from people personally impacted by the program. We wondered if having this many choices could result in decision fatigue and cause the reader to not engage with ANY of the options. We developed a treatment to test this theory and isolated the choice to read a single article. We believed the treatment would help the reader have clarity by establishing a clear call to action. We split the audience 50/50 and measured the impact on clicks.
Research Question
We believe that presenting a single call to action for cultivation message readers will achieve an increase in engagement because the isolated option will reduce potential friction caused by decision fatigue.
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.21% | ||
| T1: | Treatment #1 | 0.41% | 95.8% | 100.0% |
This experiment has a required sample size of 6,011 in order to be valid. Since the experiment had a total sample size of 329,829, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
95.8% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
What we discovered through this experiment was the treatment resulted in a 95.8% increase in clicks with a 100% statistical level of confidence. The key learning from this experiment is that presenting a single call to action for cultivation message readers can significantly increase engagement by reducing decision fatigue. This means that the control copy that offered a full menu of options may have overwhelmed readers and resulted in lost engagement opportunities. In future experimentation, it is recommended to simplify the call to action and reduce options to reduce the risk of decision fatigue and improve engagement rates.