How different types of digital advertising content affects direct mail revenue Experiment ID: #8434

Hillsdale College

Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.

Experiment Summary

Timeframe: 11/12/2018 - 1/31/2018

Hillsdale College had run an insightful experiment during the summer that showed that online advertising could have a significant impact on direct mail results. They wanted to run a subsequent test to accomplish two objectives. First, they wanted to see how the same principle affected a different package at a different time of year. Second, they wanted to see if showing different types of content affected revenue. To accomplish this objective, they created five unique segments: a control, that would see no ads, and four segments that would see Imprimis ads, Online Courses ads, Brand ads, and a mix of ads, respectively.

They set up each ad group and segment and showed ads one week before the mail packages hit homes and two weeks after to determine the results.

Research Question

How will different types of digital advertising content affect direct mail revenue?


Treatment Name Revenue per Visitor Relative Difference Confidence Average Gift
C: Control $2.05 $58.13
T1: Imprimis Ads $1.99 -3.0% 74.3% $56.95
T2: Online Courses Ads $1.89 -7.5% 95.4% $57.08
T3: Brand Ads $2.04 -0.1% 51.1% $59.53
T4: Mix $2.01 -2.0% 66.5% $58.38

This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was met so these experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 5.8% decrease in conversion rate
× 1.8% decrease in average gift
7.5% decrease in revenue

Key Learnings

Only one of the groups showed any difference in significance—the online courses group, which showed a 7.5% decline in revenue. Overall, the results were a big change from what the earlier test had shown. There are a few potential hypotheses as to this outcome:

1. It is difficult to affect results when motivation is higher. Therefore, the package that dropped closer to year-end had an overall higher average conversion rate, and thus the ads did not make much of an impact.
2. The online courses segment suffered in revenue because people signed up for (and contributed to) online courses, thus reducing their propensity to give.
3. The content of the direct mail piece was stronger, which led to a higher conversion rate and less variance between the packages.

This shows that this test should be replicated with different direct mail pieces at different times of year and with different content—especially acquisition mailings.

Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.

Experiment Documented by...

Jeff Giddens

Jeff is the President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.