How a simplified newsletter affects click rate over a templatized version, part II Experiment ID: #18421

Dallas Theological Seminary

The DTS mission is, “to glorify God by equipping godly servant-leaders for the proclamation of His Word and the building up of the body of Christ worldwide.” They strive to help men and women fulfill the Great Commission and the Great Commandment, or more simply: Teach Truth. Love Well.

Experiment Summary

Ended On: 10/8/2019

In a previous test, DTS was sending out their top three DTS Voice articles to their subscribers as a cultivation opportunity, and wanted to see whether a simplified template (links only in bullet points) or a more designed layout, with imagery and CTA’s along with blurbs about the content, performed better. They used the same content pieces in each email.

After the original test, we hypothesized that that the CTA buttons were potentially causing friction more than the design of the newsletter, and that that was the cause of the original lift from the second version.

In this test, we tried linked text in the templatized version vs. the original simplified template.

Research Question

Which newsletter template gets more clicks?

Design

C: Templatized Newsletter with Linked Text
T1: Simplified Newsletter

Results

Treatment Name Click Rate Relative Difference Confidence
C: Templatized Newsletter with Linked Text 2.2%
T1: Simplified Newsletter 4.0% 84.2% 100.0%

This experiment has a required sample size of 684 in order to be valid. Since the experiment had a total sample size of 138,460, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    84.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

Even with the adjustments made to the templatized version, the simple newsletter still received 82% more clicks than the control. Moving forward, we’ll use the simple template to get this content to DTS Voice subscribers in the most effective way.


Experiment Documented by...

Allison Jones

Allison is an Optimization Associate at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.