How a simplified newsletter affects click rate over a templatized version
Dallas Theological Seminary
The DTS mission is, “to glorify God by equipping godly servant-leaders for the proclamation of His Word and the building up of the body of Christ worldwide.” They strive to help men and women fulfill the Great Commission and the Great Commandment, or more simply: Teach Truth. Love Well.
Experiment Summary
Ended On: 10/08/2019
DTS was sending out their top three DTS Voice articles to their subscribers as a cultivation opportunity, and wanted to see whether a simplified template (links only in bullet points) or a more designed layout, with imagery and CTA’s along with blurbs about the content, performed better. They used the same content pieces in each email.
Research Question
Which newsletter template gets a higher click rate?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Templatized Newsletter | 2.2% | ||
| T1: | Simplified Newsletter | 4.0% | 77.2% | 100.0% |
This experiment has a required sample size of 773 in order to be valid. Since the experiment had a total sample size of 117,403, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
77.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The simplified version of the test increased click rate by 77%. This Isn’t surprising considering almost every test we’ve run that puts a stripped down template up against a design-heavy template delivers the same result- however, we had a hypothesis that the CTA buttons were potentially causing friction more than the design of the newsletter. We ran a subsequent test with the September newsletter to test this theory.