How versioning ad copy to lapsed donors affects conversion rate
The Heritage Foundation
Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.
Experiment Summary
Ended On: 12/18/2023
The Heritage Foundation was showing ads to lapsed donors to reactivate them to active donors. However, these ads were originally developed for acquisition. They wanted to test the impact of adding “acknowledgement” language of the donors’ past giving history to the ad copy, with the thought that it might personalize the copy and make the donor realize that they were being spoken to directly.
They launched a test against the control language to determine a winner.
Research Question
We believe that versioning copy to acknowledge support for past donors will achieve an increase in conversion rate because it will personalize the ask and acknowledge their previous behavior.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.03% | ||
| T1: | Treatment 1 | 0.01% | -75.7% | 97.1% |
This experiment has a required sample size of 23,539 in order to be valid. Since the experiment had a total sample size of 67,500, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 75.7% decrease in conversion rate
× 0% increase in average gift
Key Learnings
The key learning from this experiment is that adding “acknowledgement” language to the ad copy for lapsed donors did not lead to an increase in conversion rate. In fact, it resulted in a significant decrease in donations compared to the control group.
Based on this outcome, it is clear that personalizing the ask and acknowledging past support did not resonate with lapsed donors in this particular campaign. Therefore, it is important for future experimentation to focus on other strategies or messaging approaches to reactivate lapsed donors.
It is also important to consider the statistical significance of the results. With a 97.1% level of confidence, we can be highly confident that the decrease in donations is not due to random chance. This strengthens the validity of the findings and emphasizes the importance of using the control group as a benchmark for comparison.
In future experimentation, it would be beneficial to explore alternative tactics for re-engaging lapsed donors. This could involve testing different messaging techniques, creative approaches, or targeting strategies. It is also essential to continue tracking and analyzing key metrics such as conversion rate, donation amount, and donor retention to assess the effectiveness of any changes made.