The Heritage Foundation

How a recurring tab impacted conversions

Experiment ID: #167827

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Timeframe: 01/10/2024 - 01/26/2024

Pathway to Victory is the broadcast ministry of First Dallas.

PTV’s offers are primarily various books and premiums with donations. Our January emphasis was to increase the number of recurring donors. With this in mind, we decided to try adding a recurring tab to normal offers to see if we could passively acquire recurring donors by just having the option available.

Research Question

We believe that by adding a monthly option for web donors will achieve higher conversion rate because there is additional give options and clarity in the type of gift.

Design

C: Control
T1: Variation 1

Results

 Treatment NameConv. RateRelative DifferenceConfidenceAverage Gift
C: Control 4.1%$23.12
T1: Variation 1 3.5%-15.1% 76.5%$21.98

This experiment has a required sample size of 7,200 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

Note: treatment generated 0 recurring gifts

The key learning from this experiment is that adding a monthly option for web donors did not achieve a higher conversion rate. In fact, the treatment group saw a decrease in transactions compared to the control group. However, it is important to note that the statistical level of confidence is below 85%, meaning the results are not considered valid and should not be used for future decision-making.

Based on this, it is necessary to conduct further experiments with a larger sample size or different variations to determine the effectiveness of adding a recurring tab for web donors. It is also important to consider other factors that may have influenced the results, such as the timing of the experiment or the specific offer being presented.

Future experimentation should focus on gathering more data and conducting rigorous A/B testing to determine the impact of different variables on conversion rates. It may also be beneficial to analyze other aspects of the donor journey, such as the effectiveness of different messaging or marketing channels, to gain a deeper understanding of donor behavior and preferences. Additionally, seeking feedback from donors and analyzing their motivations and decision-making processes can provide valuable insights for future experiments.


Experiment Documented by NextAfter

Question about experiment #167827

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.