How including an image in a personal email affects donor conversion
Alliance Defending Freedom
Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.
Experiment Summary
Timeframe: 03/01/2017 - 03/10/2017
Alliance Defending Freedom was raising money for a very specific case. They were renting email lists for the first time, and so they knew that, to many of the recipients, they were introducing the person they were defending for the first time. However, since this case made national news, they knew that many of the recipients would be familiar with the case, if not the name of the defendant.
They had tested several different treatments and had arrived at a very plain text email with no formatting. However, since they were introducing someone new, they thought that it might produce a lift if they included a picture of Barronelle, the defendant. Since this was truly an introduction, then they thought the reader might be more likely to respond if they could see her before they reached the landing page, and when they reached the landing page, they might be more likely to convert.
They created a treatment with an image and split the next email list rental in half to determine a winner.
Research Question
Will including an image in a personal email increase donor conversion?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | No Image | 0.05% | ||
| T1: | With image | 0.09% | 80.8% | 98.6% |
This experiment has a required sample size of 31,771 in order to be valid. Since the experiment had a total sample size of 100,000, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 80.8% increase in conversion rate
× 0% increase in average gift
Key Learnings
The treatment with the image had a significant effect—not only a big lift in clickthrough rate, but also a big lift in donations—80.8%.
This illustrates the power of imagery as essential to understanding the value proposition expressed in the email. While a previous email template had proven to reduce conversion, it was because the imagery was not relevant to the value proposition and instead added unnecessary friction. However the image of Barronelle in this evoked a powerful emotion—in this case, sympathy—which increased the force of the value proposition and even triggered a higher motivation to continue through with the donation.
This treatment was quickly adopted as the new control, and further testing began to try to personalize it even more without losing the directness of the message.