Impact of smart content on a Fiscal Year End Campaign Donation Page
Care Net
Experiment Summary
Timeframe: 06/17/2020 - 07/02/2020
After our last experiment testing the effectiveness of smart content on a campaign donation page (Experiment 30207), we decided to test this again during a fiscal year end campaign. We wanted to see if the greater motivation and urgency would make both pages return similar conversion rates.
Research Question
Will smart content
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Raise Donors (non-smart content) donation page | 45.0% | ||
| T1: | DepositFix (smart content) donation page | 25.8% | -42.7% | 100.0% |
This experiment has a required sample size of 47 in order to be valid. Since the experiment had a total sample size of 500, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 42.7% decrease in conversion rate
× 0% increase in average gift
Key Learnings
While our first experiment increased conversion by 50%, this experiment decreased donor conversion by almost the same amount. Moreover, each of the three individual emails brought more donations when people clicked to the non-smart content page than when they went to the control page. The smart content page did not win on any of the emails.
We are looking at doing one additional test to see what happens if we mimic the “look” of the control page better on the variation. We want to ensure that this “look” difference isn’t dragging down conversion.