How a templatized cultivation email affects click rate vs. a simple format Experiment ID: #18204

Dallas Theological Seminary

The DTS mission is, “to glorify God by equipping godly servant-leaders for the proclamation of His Word and the building up of the body of Christ worldwide.” They strive to help men and women fulfill the Great Commission and the Great Commandment, or more simply: Teach Truth. Love Well.

Experiment Summary

Timeframe: 8/29/2019 - 9/4/2019

DTS was sending out a monthly DTS Voice newsletter, and wanted to test the format of the email to see whether more people clicked links in a templatized format with CTA’s, or a simpler version with bullet pointed links.

Research Question

Do more people click links in a templatized cultivation email or a simpler one?


C: Templatized
T1: Simple


Treatment Name Click Rate Relative Difference Confidence
C: Templatized 2.0%
T1: Simple 3.6% 77.8% 100.0%

This experiment has a required sample size of 840 in order to be valid. Since the experiment had a total sample size of 117,421, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    77.8% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The simpler version of the email produced significantly more click engagement than the templatized version. We have a hypothesis that the use of CTA buttons in the first email contributed to the lower click rate, so future testing will involve linked text instead of buttons on the templatized version.

Experiment Documented by...

Allison Jones

Allison is an Optimization Associate at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.