The Heritage Foundation

How explaining the value of the membership card to non-members in a fundraising email affects the click-through rate

Experiment ID: #173541

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Ended On: 03/27/2024

We conducted this email experiment for The Heritage Foundation’s Members Card campaign. The email asked non-members to give, in return, they would receive a Members Card. The email also announces a matching gift opportunity. The conclusion of the email includes a call to action to donate.

We wanted to know if non-members would click more if they were reminded about the value they receive when they donate. The treatment included additional text towards the beginning of the email which explained the value proposition.

Research Question

Including reminder text about the value of a membership card will increase the open rate because it is an additional opportunity to explain the value non-members receive.

Design

C: Control
T1: Treatment #1

Results

 Treatment NameClick RateRelative DifferenceConfidence
C: Control 0.29%
T1: Treatment #1 0.29%0.6% 11.0%

This experiment has a required sample size of 75,123,367 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

The key learning from this experiment is that including reminder text about the value of a membership card can potentially increase the click-through rates for donation emails, as evidenced by the 0.6% increase in Clicks for All Traffic in the treatment group. However, the statistical significance level of 11.0% is below the standard threshold of 85%, indicating that these results may not be reliable for future decision-making.

Note: While the click-through rate increased in the treatment email, the conversion rate had a slightly lower conversion rate.

Moving forward, it is important to continue exploring the impact of including value proposition reminders in donation appeals, perhaps refining the messaging or placement within the email to maximize effectiveness. Additionally, it would be beneficial to conduct further experiments with larger sample sizes to validate the findings and make more informed decisions for future campaigns.


Experiment Documented by NextAfter

Question about experiment #173541

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.