How framing the opposition in familiar terms affects donor conversion Experiment ID: #16213

Leadership Institute

Experiment Summary

Ended On: 5/29/2019

The Leadership Institute has long raised money to train conservatives to beat “leftists.” This term was chosen over the more common “liberal” since “leftist” was believed to be more specific, more powerful, and convey more emotion. Moreover, “liberal” could be imprecise, referring in some contexts to “classical liberals” who often align with conservatives.

They wondered, though, whether these assumptions were correct. Is “liberal” in fact a better understood term than “leftist,” and might it convey more emotion?

They decided to run a test to see if the more commonly used “liberal” would increase response. Given relatively small sample sizes, they knew it would take several email appeals to determine a winner, and they also knew they didn’t want to validate it on a single email. They set up three email appeals, with “liberal” and “leftist” being the only words they interchanged.

Research Question

How will changing the terminology that frames the opposition affect donor conversion?


C: Leftist
T1: Treatment #1


Treatment Name Conv. Rate Relative Difference Confidence
C: Leftist 0.06%
T1: Treatment #1 0.13% 119.3% 97.9%

This experiment has a required sample size of 14,998 in order to be valid. Since the experiment had a total sample size of 41,660, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 119.3% increase in conversion rate
× 0% increase in average gift

Key Learnings

The appeals that used the term “liberal” produced 119.3% more donors than the “leftist” appeals. This is fascinating, because the change was so small, yet generated such a different response in the mind of the donor. This suggests that “liberal” is closer to how the donor natively frames the opposition—or, it suggests that the donor believes that the “liberal” is of some greater threat than the “leftist”. Either way, this shows that a small change in terminology and framing of the opponent can have a big impact to the bottom line.

Experiment Documented by...

Jeff Giddens

Jeff is the President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.