How different types of digital advertising content affects direct mail revenue Experiment ID: #8434
Founded in 1844, Hillsdale College is an independent liberal arts college with a student body of about 1,400. Hillsdale’s educational mission rests upon two principles: academic excellence and institutional independence. The College does not accept federal or state taxpayer subsidies for any of its operations.
Timeframe: 11/12/2018 - 1/31/2018
Hillsdale College had run an insightful experiment during the summer that showed that online advertising could have a significant impact on direct mail results. They wanted to run a subsequent test to accomplish two objectives. First, they wanted to see how the same principle affected a different package at a different time of year. Second, they wanted to see if showing different types of content affected revenue. To accomplish this objective, they created five unique segments: a control, that would see no ads, and four segments that would see Imprimis ads, Online Courses ads, Brand ads, and a mix of ads, respectively.
They set up each ad group and segment and showed ads one week before the mail packages hit homes and two weeks after to determine the results.
How will different types of digital advertising content affect direct mail revenue?
|Treatment Name||Revenue per Visitor||Relative Difference||Confidence||Average Gift|
|T2:||Online Courses Ads||$1.89||-7.5%||95.4%||$57.08|
This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was met so these experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 5.8% decrease in conversion rate
× 1.8% decrease in average gift
7.5% decrease in revenue
Only one of the groups showed any difference in significance—the online courses group, which showed a 7.5% decline in revenue. Overall, the results were a big change from what the earlier test had shown. There are a few potential hypotheses as to this outcome:
1. It is difficult to affect results when motivation is higher. Therefore, the package that dropped closer to year-end had an overall higher average conversion rate, and thus the ads did not make much of an impact.
2. The online courses segment suffered in revenue because people signed up for (and contributed to) online courses, thus reducing their propensity to give.
3. The content of the direct mail piece was stronger, which led to a higher conversion rate and less variance between the packages.
This shows that this test should be replicated with different direct mail pieces at different times of year and with different content—especially acquisition mailings.