Reasons to Believe

How giving is impacted by the use of buttons in an email appeal

Experiment ID: #123471

Reasons to Believe

Experiment Summary

Ended On: 01/17/2023

RTB historically uses multiple “Give Now” buttons in email appeals. During calendar year end we decided to run an experiment where we removed the buttons from the email appeal. The idea is that the removal of the buttons would actually humanize the communication more, make the email seem less promotion-like, and encourage people to read the entire appeal/case for support before deciding to give. This in turn, has the potential to increase donor conversion rate. We chose one email in the campaign to test this concept.

Research Question

We believe that removing donation buttons from an email appeal for email recipients will achieve a higher donor conversion rate .

Design

C: Control
T1: Treatment #1

Results

  Treatment Name Conv. Rate Relative Difference Confidence
C: Control 0.12%
T1: Treatment #1 0.12% -0.4% 0.7%

This experiment has a required sample size of 518,499,973 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

Results in this experiment were flat. We saw .35% decrease in donor conversion rate with a .68% level of confidence. The average gift across both the control and treatment were also negligible (control: $250, treatment: $238) so the humanized treatment did not inspire people to give more either. A concern in removing the buttons from the email was that it would negatively impact click thru rate. However, results indicate that the removal of the buttons has no effect on click thru rate either. Click thru rate stayed the same at .4%.

Where we did see a noticeable difference was the impact on new donors. New donors experienced a 497% increase in donor conversion rate with a level of confidence of 94%. More non donors were activated to give through the treatment (no buttons) than the control. While it’s not a large number of transactions, the organization should consider using this new layout (no buttons) knowing that it will not impact giving overall but may active more new donors. It’s likely that non donors saw the email as less transactional/promotional and decided to read more of the appeal and were therefore more motivated to give.


Experiment Documented by NextAfter

Question about experiment #123471

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.