How adding benefits of monthly giving affected clicks in an appeal | NextAfter
Texas Public Policy Foundation

How adding benefits of monthly giving affected clicks in an appeal

Experiment ID: #53756

Texas Public Policy Foundation

Experiment Summary

Timeframe: 03/03/2021 - 03/11/2021

For a high-urgency campaign, we wanted to test if including the benefits of becoming a monthly donor would increase clicks to the donation page. This was the last email in the campaign and it was set up a personal note forwarding the previous email from earlier in the day with one final request to give before the deadline. The treatment version listed the benefits of becoming a monthly donor and included an image of the premium for monthly donors.

Research Question

We believe that adding benefits of monthly giving for a housefile appeal will achieve a higher click rate.

Design

C: Version A
T1: Version B

Results

Treatment Name Click Rate Relative Difference Confidence
C: Version A 0.44%
T1: Version B 0.55% 24.1% 43.8%

This experiment has a required sample size of 33,539 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

Since this email was sent to an audience limited in size, we were unable to validate the results of the experiment. We saw a 24.1% increase in traffic on the treatment but with only a 43.8% level of confidence so we cannot claim statistical significance. Moving forward, we’ll plan to retest this same concept in future emails—as well as launch the test in other channels that may reach a wider audience —to see if there are any learnings to be applied.


Experiment Documented by Rebekah Josefy
Rebekah Josefy is an Optimization Director at NextAfter.

Question about experiment #53756

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.