How both a campaign slidedown and slideout impacts email acquisition rate on blog pages
Alliance Defending Freedom
Alliance Defending Freedom is an alliance-building legal organization that advocates for the right of people to freely live out their faith.
Experiment Summary
Timeframe: 10/30/2017 - 11/21/2017
Alliance Defending Freedom recently found that adding a campaign donation slidedown to their homepage increased donor conversion by nearly 500%. As they continued to introduce this approach across their site, they decided to isolate traffic to their blog pages and conduct a similar test. This time they wondered: Will campaign slidedowns and slideouts that encourage the site visitor to sign up for updates on the Jack Phillips case increase email acquisition rate? They created a treatment version that included these campaign elements and launched an A/B test to find out.
Research Question
Will a campaign-specific slidedown and slideout increase email acquisition rate on blog pages?
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | No Interrupters | 0.03% | ||
| T1: | Email Interrupters | 0.70% | 2,001.4% | 100.0% |
| T2: | Donation Interruptors | 0.04% | 11.9% | 14.7% |
This experiment has a required sample size of 655 in order to be valid. Since the experiment had a total sample size of 46,408, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 2,001.4% increase in conversion rate
× 0% increase in average gift
Key Learnings
Alliance Defending Freedom found that the treatment version increased email acquisition by 2,000%. It’s clear these campaign-specific slidedowns and slideouts were effective at converting website traffic into motivated fans and potential donors of ADF. While this test is supported by many of the learnings produced in the prior homepage slidedown test we discussed earlier, there are some key differences and new findings worth mentioning. First, it’s important to differentiate between the donation slidedown and the email slidedown and slideout. While the donation slidedown used a CTA button to drive traffic to a ‘Jack Phillips’ donation page, the email acquisition slidedown and slideout contained a email form with no clickthrough opportunity. By removing a step in the visitor’s pathway (i.e. keeping the visitor on the same page), we found that this version reduced friction and allowed motivated visitors to take action immediately by signing up for updates. We can use this tactic here, since the level of commitment for a donation is higher than an email address and requires more information.
The addition of the campaign slideout also deserves credit for this significant increase. On the homepage slidedown test, visitors were only given the option of clicking through to the donation page at the very top of the site. If they scrolled down quickly, they would not see this option again. In this email acquisition version, the slidedown still served as the primary call-to-action. However, if a visitor scrolled down past 50% of the page, they would also see the slideout. This addition reintroduced the offer seen at the top of the page, adding urgency to the campaign offer.
Finally, this test takes a good step in determining the difference between the nature of homepage visitors and blog page visitors. It appears here that blog page visitors were highly motivated, when presented an email acquisition offer. However, this test also re-validated that the Alliance Defending Freedom blog is not a great direct source for donations. Further testing is being conducted to hone in on the particularities between blog visitors and homepage visitors.