Gmail’s handling of full-format vs. low-format emails

Jewish Voice Ministries

JVM’s mission is two-fold: Proclaiming the Gospel to the Jew first, and also to the Nations (Romans 1:16), and equipping the Church—providing education about the Hebraic roots of Christianity, the Church’s responsibility to Israel and the Jewish People, and how to share Messiah with the Jewish People.

Experiment Summary

Timeframe: 07/20/2017 - 08/10/2017

We had already been testing a low-format version of our emails (no tables, no images, limited CSS, visible links) and seeing that they performed several percentage points better in open rate than the fully formatted versions. We suspected that one or more email clients was treating low-format emails differently than full-format emails, and our first suspect was Google due to its tabbed inbox system.

Research Question

Does Gmail treat low-format emails differently than full-format emails?

Results

This experiment has a required sample size of 138 in order to be valid. Since the experiment had a total sample size of 76,320, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    69.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

It turns out that, with no changes to email address, subject line or preheader text, Gmail directs low-format emails into the Primary inbox and full-format emails into the Promotions tab. With the email in the Primary inbox, open rates increase by 50-60%. This has held true for nearly a dozen sends over two months.

This indicates that a vast majority of Gmail users don’t know about the Promotions tab or forget to look there. We’ve also determined that most Gmail mobile apps make the Promotions area hard to find, although Google now includes Promotions notifications in the Primary inbox.

As a further note, no other email client shows a statistically significant improvement from a low-format version, and AT&T properties show a negative impact.