How adding benefits of monthly giving affected clicks in an appeal
Texas Public Policy Foundation
Experiment Summary
Timeframe: 03/03/2021 - 03/11/2021
For a high-urgency campaign, we wanted to test if including the benefits of becoming a monthly donor would increase clicks to the donation page. This was the last email in the campaign and it was set up a personal note forwarding the previous email from earlier in the day with one final request to give before the deadline. The treatment version listed the benefits of becoming a monthly donor and included an image of the premium for monthly donors.
Research Question
We believe that adding benefits of monthly giving for a housefile appeal will achieve a higher click rate.
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Version A | 0.44% | ||
| T1: | Version B | 0.55% | 24.1% | 43.8% |
This experiment has a required sample size of 33,539 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
Since this email was sent to an audience limited in size, we were unable to validate the results of the experiment. We saw a 24.1% increase in traffic on the treatment but with only a 43.8% level of confidence so we cannot claim statistical significance. Moving forward, we’ll plan to retest this same concept in future emails—as well as launch the test in other channels that may reach a wider audience —to see if there are any learnings to be applied.