31.5% lift How adding value proposition copy to a recurring gift donation page impacts conversion

Date Added: January 16, 2018 Research Partner: Focus on the Family Element tested: Donation Page Headline, Donation Page Copy

The Focus on the Family recurring gift donation page had copy on it that was very transaction focused. It had been that way for years. They hypothesized that they might be able to increase conversion if the copy was more value proposition focused rather than transactional. They developed a treatment that told the reader what their monthly gift would do and test this against their control.

View the experiment »


Not Valid How a radical redesign for a year-end donation page impacts conversion

Date Added: January 2, 2018 Research Partner: Focus on the Family Element tested: Donation Page Design, Donation Page Headline, Donation Page Copy, Donation Page Form

During the Focus on the Family year-end campaign, we hypothesized that we might be able increase conversion by testing their current year-end donation page with a radical redesign. The radical redesign included removing the video on the page, adding a countdown clock on the page, increasing value proposition and added a giving thermometer.

View the experiment »


-98.3% drop Discovering the ideal offer to fit the motivation of the visitors

Date Added: January 2, 2018 Research Partner: Focus on the Family Element tested: Advertising

Through a series of previous experiments, we had found that an in-article advertisement for a related assessment moved to the bottom of the article, would yield a significant boost in advertising clicks. As part of an iterative testing plan, we next wanted to discover what type of offer would be most appealing to the audience.

We decided to test the assessment-offer against a product-offer that they could receive for a gift of any amount. Now, we knew that we would receive less conversions but a donor conversion would be more valuable than an email conversion. The question was, what is the trade-off between the two offers; i.e. how many donors could we acquire vs. how many email addresses could we acquire?



View the experiment »


-18.2% drop How the addition of a countdown clock on a pop-up impacts donors conversion during year-end.

Date Added: January 2, 2018 Research Partner: Focus on the Family Element tested: Advertising

In the final days of Focus on the Family's year-end giving campaign, they wanted to make it easy for donors to give. To do this, they had a pop-up appear when you visited their website. We hypothesized that we might be able to increase motivation by creating a greater sense of urgency on this pop-up with the addition of a countdown clock. The countdown clock showed people how much time they had left to give until the end of the year. We know that urgency is a motivating factor during online giving campaigns like year-end. We split the traffic that would see each version of the pop-up and tested it.

View the experiment »


Not Valid How removing a video affects email appeal revenue

Date Added: December 14, 2017 Research Partner: Focus on the Family Element tested: Email Design, Donation Page Design

We saw a number of different factors and potential learnings at play this test.

First off, eliminating the softer CTA (“Watch Video”) in the email reduced the open-to-click rate by 34.9% in the non-video treatment.  However, there wasn’t a corresponding reduction in donations—likely suggesting that the larger audience clicking the control was less motivated to give and more motivated to watch the video.

At first glance when reviewing overall giving, we saw a 2.1% increase in conversions for the non-video treatment . . . but at a 28.8% decrease in revenue. Neither met our 95% confidence threshold, although the revenue decrease had a confidence of 90.9%.  For this reason, we scrutinized the data more closely to ensure that we didn’t miss a learning.

After reviewing the individual transactions from both variants, we found that the gift amounts had significant outliers due to $,1,000+ donations.  Knowing when to filter data is a difficult challenge of testing and optimization—but since we have repeatedly seen that higher-value gifts are less likely to be motivated by an individual element of a broad-base ask, we typically exclude these from revenue validation.  After making this adjustment, we see the smaller difference noted in the results table.

Here are a few thoughts on this result, and why it requires additional testing to validate:

  1. Response sizes are too small
    Due to a low conversion rate among both target segments, there isn’t enough data to back up the normal distribution of gift amount due to its existing variability.  This is especially true of a non-donor segment that was tested with similar results, where conversion rate was just 0.007%.  Further testing, including during higher response windows, may improve the actionability of results.
  2. The video wasn’t the only thing tested
    When removing the video thumbnail from the email, it was replaced with a different image, which was also rotated.  The additional call to action button was also removed instead of being modified. Copy was added in that area, but from a different testimonial changing the value being conveyed.On the landing page, the same additional testimonial copy was used, but was added much later in the copy compared to the video.  When testing video as a medium, the core value proposition of the video should be represented in text—if not, you’re also testing between different value proposition or the removal of some of the value proposition presented completely.
  3. There were significantly more outliers in the control segment
    While excluding $1,000+ gifts from both segments made a difference, it’s difficult to tell what the normal distribution of gifts is and whether segmentation played a factor in the results.  Typically the only ways to answer these questions are to measure historical distributions to be able identify abnormal ones, or to have a larger number of results to help balance any small number of outliers.

Due to the significant impact we’ve seen when replacing video on appeal pages, we would recommend further testing to ensure a clear and applicable learning for this audience.



View the experiment »


-3.4% drop How adding a call-to-action to the top of a simplified landing page impacts conversion

Date Added: November 30, 2017 Research Partner: Focus on the Family Element tested: Name Acquisition Design

Focus on the Family offers a Marriage Assessment on their website. In a previous test on the assessment landing page, we learned that a simplified page where we removed elements of friction actually decreased conversion because we removed a button at the top of the page that would take this highly motivated audience directly to the assessment. To build upon those learnings, we hypothesized that we could increase conversion by keeping and testing the simplified version again but simply just adding back in the button to the assessment at the top of the page. We split the traffic and tested it.

View the experiment »


101.2% lift How the placement of an offer on article pages impacts conversion

Date Added: November 29, 2017 Research Partner: Focus on the Family Element tested: Advertising, Name Acquisition Design

By placing the offer at the end of the article, we were able to increase conversion by 101%! This tells us a couple of things:

1. Placement off an offer on a page matters significantly
2. People are more highly motivated to want/get the offer after they have experienced the full value of the content on the page. The offer acts as a natural next step in the mind of the end-user.

Now that we know where to place offers within article pages to get the highest conversion, we recommend testing the kind of offer we present in that space.



View the experiment »


-74.9% drop How an inline ask in place of a midroll ad affects clickthrough

Date Added: November 9, 2017 Research Partner: Focus on the Family Element tested: Advertising

The inline ask saw a 74.9% decrease in clicks compared to the normal banner ads.  While there were broader appeal offers mixed into the ads that makes the comparison indirect, we hypothesize from low donor conversion rate that the original motivation of this audience may not align well with a direct donation ask.  This result stands in stark contrast to the results of our similar test of an inline email acquisition offer in this same placement.

We’re reminded that motivation is both the strongest factor of conversion as well as the most difficult to influence.  We plan to continue testing the placement and type of offers used on these long-form article pages to improve alignment with the content and visitor motivation.



View the experiment »


1,870.7% lift How inline acquisition offers in place of ads affected clickthrough

Date Added: November 9, 2017 Research Partner: Focus on the Family Element tested: Advertising, Name Acquisition Design

Focus on the Family currently uses DoubleClick for Publishers to manage ad fill throughout their website.  Most of the ads in rotation are for their own resources, giving opportunities, and some third-party offers. One such placement is a mid-roll banner targeting long-form article pages.  They hypothesized that an inline email acquisition offer that is contextually relevant for their audience may increase clickthrough and help capture more of the high volume of new visitors viewing these articles. They already had a free online Marriage Assessment which has resonated well with visitors to their marriage and parenting content, so they split the traffic to articles in those categories and delivered an inline offer for the assessment in place of the mid-roll ad from DoubleClick to half the audience.

View the experiment »


5.5% lift How friction and anxiety influence donor conversion

Date Added: November 8, 2017 Research Partner: Focus on the Family Element tested: Donation Page Design, Donation Page Form

The treatment resulted in a 5.5% increase to donor conversion. While it is a relatively small percentage lift, since this test impacts all donation forms, it will result in a six-figure increase in revenue in the coming 12 months.

When doing a deeper analysis of the experiment, we found that the new treatment did not have as significant of an impact on traffic driven by email but was significantly more impactful for mobile traffic. For the mobile segment, it led to a 9.8% increase to overall conversion.

What this suggests is that the reduction of friction is not as important to a highly motivated audience (like email.) But for organic site visitors or those browsing on their phone, making the gift easier, was able to positively influence this lower motivated traffic.



View the experiment »