How a radical change to a main donation page affects donor conversion
e3 Partners
Experiment Summary
Timeframe: 04/05/2022 - 05/23/2022
I Am Second is a ministry that uses film to capture the true and authentic story of individuals whose lives have been changed when they put Jesus first in their lives. I Am Second has huge brand awareness due in large part to the celebrity nature of the film subjects, but they do not put a large focus on fundraising due to legacy donor support.
Usually, NextAfter suggests making a singular change to a control in order to isolate lift and motivation. In this case, we decided to make a radical change to the main donation page to test the removal of a variety of items that we feel are causing friction. Here is a summary of what was changed:
- We swapped out donation platforms (keeping the donation form fill on a singular page)
- Changed the headline to align with the organizational value proposition
- Removed the navigational header
- Changed the body copy to be more donor-centric, vs org centric
- Removed footer elements
It is important to note that the action being tested and tracked was the donate button. As these were two different donation platforms (due to challenges with making changes to the donation pathway testing), we wanted to isolate the actions to just button clicks.
Research Question
We believe that removing friction from the donation experience for potential donors will achieve a greater number of donor conversions.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Control | 0.00% | $0.00 | ||
| T1: | iDonate Treatment | 8.7% | 100.0% | 96.4% | $0.00 |
This experiment has a required sample size of 40 in order to be valid. Since the experiment had a total sample size of 97, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 100.0% increase in conversion rate
× 0% increase in average gift
Key Learnings
While I am Second does have campaign-specific donor appeals that generate a decent amount of traffic to their campaign-specific donation pages, their main donation page has always been secondary to the overall goal. As such, it does not receive much traffic, and therefore sample sizes can be difficult to achieve validity. However, in this case, we were able to see an 8.7% increase in donor conversions (the donate button click) with a 96.3% level of confidence.
Key takeaways: While sample sizes are low, we still reached validity in the experiment. However, since we are changing so many things radically, it would be best to work backwards, seeing what we could add back into the page w/out creating more friction. Also, since we used two different donation platforms, it would be interesting to see if we could rerun the experiment on one platform.
As I Am Second is undergoing an entire site redesign, they plan on using the learnings from this experiment to guide changes to the new main donation page.
Additional observations of the impact on other visitor segments in the experiment:
- Returning visitors had a 14.3% increase in donations with a 94% level of confidence.
- Mobile visitors had a 9.7% increase in donations with a 93% level of confidence.