How changing the location of the argument affects clicks
Federation for American Immigration Reform
Experiment Summary
Timeframe: 09/10/2014 - 09/12/2014
As part of their September appeal, FAIR was using the results of their most recent study to demonstrate the need for greater immigration reform. Their Control email took the standard approach of demonstrating the argument, both the problem and solution) within the email while making a direct ask for support at the bottom of the email. As an alternative to this approach, we developed a new treatment that changed the look and feel of the email to look more personal and only discussed the problem within the email.
Research Question
Does a more personalized email with content that describes the problem but not the solution generate more clicks than a more heavily designed email that is geared towards driving visitors to engage with the information on the website?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Designed Email with Interactive Map | 3.0% | ||
| T1: | Simple Email with Clickbait Language | 5.2% | 70.9% | 100.0% |
This experiment has a required sample size of 653 in order to be valid. Since the experiment had a total sample size of 43,676, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
70.9% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
When presenting both the problem and solution within the email, the visitors did not have the same motivation to visit the website. The new treatment was able to drive 70.9% more visitors to the website in order to learn about FAIR’s solution to the problem presented within the email.