How promoting a free ebook offer instead of a general subscription impacted email acquisition on a commentary page
The Gospel Coalition
Experiment Summary
Timeframe: 10/18/2022 - 11/18/2022
The Gospel Coalition has a Commentary page in which visitors can study and learn about different books of the Bible. Towards the bottom of the page, there is an email subscription form where visitors can sign up for email updates. We wondered if this existing email acquisition opportunity could be optimized to be more alluring and enticing. So we developed a treatment that offered a free eBook for visitors when signing up and submitting their email. And the offer isn’t just for any eBook, but for TGC’s most popular eBook. Would this opportunity acquire more emails than the original opportunity?
We tested this on the TGC home page and saw negative results for the treatment; now we’re testing it on the Commentaries page to see if it aligns better when implemented there.
Research Question
We believe that offering a free eBook at the bottom of the Commentary page for page visitors when submitting their email will achieve an increase in email acquisition because they will be more drawn to this offer after reading gospel commentary than they would be receiving email updates from TGC.
Design
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Content Offer | 0.00% | $0.00 | ||
| T1: | Content Offer - New copy | 0.92% | 100.0% | 100.0% | $0.00 |
This experiment has a required sample size of 421 in order to be valid. Since the experiment had a total sample size of 10,291, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 100.0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The treatment (eBook offer) saw an increase in email acquisitions across all segments. Overall, the treatment produced a 0.92% increase in email acquisitions at a 99.9% level of confidence. The control acquired zero emails for the duration of the test. The results tell us that page visitors were more excited to receive a free eBook than they were to sign up and receive email updates from TGC.
It’s worth restating that we ran this same test on the TGC home page and the treatment saw negative results. The difference is people often go to the home page to sign up for email updates and they expect an opportunity to sign up for email updates to exist on the home page. That is not the case for TGC’s Commentary page. People visit this page to acquire specific knowledge and understanding. Thus, as they keep scrolling down the page, they’re intrigued by an offer that also offers them the chance to acquire knowledge and understanding. They don’t care at this stage to sign up for general email updates.
This concept is worth testing on pages for which the free offer aligns with the page content. It would be worth testing on other content-heavy pages.