How replacing imagery with content impacts donor conversion from email
American Cornerstone Institute
Experiment Summary
Ended On: 06/29/2021
As American Cornerstone Institute was launching their new organization, they utilized a previously established email file that was created during Dr. Carson’s presidential run. These subscribers had a prior affinity for the ACI Chairman which would indicate they were politically conservative.
In launching their organization, ACI secured an endorsement from President Trump and wanted to promote this to the email file as a way to get people to support the new organization. However, we had the hypothesis that focusing on the value proposition behind the organization would be more impactful than just the single quote. We decided to test this with two treatments: one that expanded on the reason behind ACI and one that did that while also including the quote from President Trump.
Research Question
We believe that emphasizing the value proposition for email subscribers will achieve increased donations.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Image Focused | 0.02% | ||
| T1: | Value Prop w/ Trump Quote | 0.11% | 478.6% | 100.0% |
| T2: | Value Prop w/o Trump Quote | 0.12% | 518.1% | 100.0% |
This experiment has a required sample size of 9,819 in order to be valid. Since the experiment had a total sample size of 238,585, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 478.6% increase in conversion rate
× 0% increase in average gift
Key Learnings
The most significant learning was that focusing on the “why” behind ACI led to dramatically more donations (478% and 518%). We cannot rely solely on imagery of known political figures to carry the messaging. It is also worth noting that removing the President Trump quote resulted in an increase in donations. It is not statistically significant but it is something that is worth isolating in future experiments.