Impact of smart content on a Fiscal Year End Campaign Donation Page Experiment ID: #21600

Care Net

Experiment Summary

Timeframe: 6/17/2020 - 7/2/2020

After our last experiment testing the effectiveness of smart content on a campaign donation page (Experiment 30207), we decided to test this again during a fiscal year end campaign. We wanted to see if the greater motivation and urgency would make both pages return similar conversion rates.

Research Question

Will smart content

Design

C: Raise Donors (non-smart content) donation page
T1: DepositFix (smart content) donation page

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Raise Donors (non-smart content) donation page 45.0%
T1: DepositFix (smart content) donation page 25.8% -42.7% 100.0%

This experiment has a required sample size of 47 in order to be valid. Since the experiment had a total sample size of 500, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 42.7% decrease in conversion rate
× 0% increase in average gift

Key Learnings

While our first experiment increased conversion by 50%, this experiment decreased donor conversion by almost the same amount. Moreover, each of the three individual emails brought more donations when people clicked to the non-smart content page than when they went to the control page. The smart content page did not win on any of the emails.

We are looking at doing one additional test to see what happens if we mimic the “look” of the control page better on the variation. We want to ensure that this “look” difference isn’t dragging down conversion.

 


Experiment Documented by...

Scoreboard NextAfter

Scoreboard is . If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.