Gmail’s handling of full-format vs. low-format emails Experiment ID: #7473

Jewish Voice Ministries

JVM’s mission is two-fold: Proclaiming the Gospel to the Jew first, and also to the Nations (Romans 1:16), and equipping the Church—providing education about the Hebraic roots of Christianity, the Church’s responsibility to Israel and the Jewish People, and how to share Messiah with the Jewish People.

Experiment Summary

Timeframe: 7/20/2017 - 8/10/2017

We had already been testing a low-format version of our emails (no tables, no images, limited CSS, visible links) and seeing that they performed several percentage points better in open rate than the fully formatted versions. We suspected that one or more email clients was treating low-format emails differently than full-format emails, and our first suspect was Google due to its tabbed inbox system.

Research Question

Does Gmail treat low-format emails differently than full-format emails?

Results

Treatment Name Open Rate Relative Difference Confidence
C: Full-format email sent to Gmail 12.9%
T1: Low-format email sent to Gmail 21.9% 69.2% 100.0%

This experiment has a required sample size of 138 in order to be valid. Since the experiment had a total sample size of 76,320, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    69.2% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

It turns out that, with no changes to email address, subject line or preheader text, Gmail directs low-format emails into the Primary inbox and full-format emails into the Promotions tab. With the email in the Primary inbox, open rates increase by 50-60%. This has held true for nearly a dozen sends over two months.

This indicates that a vast majority of Gmail users don’t know about the Promotions tab or forget to look there. We’ve also determined that most Gmail mobile apps make the Promotions area hard to find, although Google now includes Promotions notifications in the Primary inbox.

As a further note, no other email client shows a statistically significant improvement from a low-format version, and AT&T properties show a negative impact.


Share this research with a colleague

Our mission is to help elevate the field of fundraising by openly sharing our research and inspiring a wider community of testing and optimization. If you have found our research to be helpful, insightful, or even just interesting—please share it with a fellow fundraiser.






Experiment Documented by...

Mark Santiago

Mark is . If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.