How framing the opposition in familiar terms affects donor conversion
Leadership Institute
Experiment Summary
Ended On: 05/29/2019
The Leadership Institute has long raised money to train conservatives to beat “leftists.” This term was chosen over the more common “liberal” since “leftist” was believed to be more specific, more powerful, and convey more emotion. Moreover, “liberal” could be imprecise, referring in some contexts to “classical liberals” who often align with conservatives.
They wondered, though, whether these assumptions were correct. Is “liberal” in fact a better understood term than “leftist,” and might it convey more emotion?
They decided to run a test to see if the more commonly used “liberal” would increase response. Given relatively small sample sizes, they knew it would take several email appeals to determine a winner, and they also knew they didn’t want to validate it on a single email. They set up three email appeals, with “liberal” and “leftist” being the only words they interchanged.
Research Question
How will changing the terminology that frames the opposition affect donor conversion?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Leftist | 0.06% | ||
| T1: | Treatment #1 | 0.13% | 119.3% | 97.9% |
This experiment has a required sample size of 14,998 in order to be valid. Since the experiment had a total sample size of 41,660, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 119.3% increase in conversion rate
× 0% increase in average gift
Key Learnings
The appeals that used the term “liberal” produced 119.3% more donors than the “leftist” appeals. This is fascinating, because the change was so small, yet generated such a different response in the mind of the donor. This suggests that “liberal” is closer to how the donor natively frames the opposition—or, it suggests that the donor believes that the “liberal” is of some greater threat than the “leftist”. Either way, this shows that a small change in terminology and framing of the opponent can have a big impact to the bottom line.