Formatted vs. Unformatted Experiment ID: #6930
Jewish Voice Ministries
JVM’s mission is two-fold: Proclaiming the Gospel to the Jew first, and also to the Nations (Romans 1:16), and equipping the Church—providing education about the Hebraic roots of Christianity, the Church’s responsibility to Israel and the Jewish People, and how to share Messiah with the Jewish People.
Timeframe: 6/29/2017 - 7/3/2017
We send a weekly email out to roughly 200,000. It comes from our CEO and is new oriented. His full name is in the FROM and the subject lines are generally like a news headline. The emails are formatted and branded.
Will stripping the formatting from an email increase the Opens, Clicks, and conversions (donations) when tested against our standard branded email?
MECLABS Conversion Factors Targeted
C = 4m + 3v + 2( i - f) - 2a ©
Copyright 2015, MECLABS
|Treatment Name||Open Rate||Relative Difference||Confidence||Average Gift|
|C:||Standard Branded Email||14.6%||$45.00|
This experiment has a required sample size of 102 in order to be valid. Since the experiment had a total sample size of 200,000, and the level of confidence is above 95%, the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
75.3% increase in traffic
× 0% increase in conversion rate
× 44.4% increase in average gift
153.2% increase in revenue
This was by far our most dramatic experiment of late, in terms of the results. Drastically more opens and clicks and decent increase in donations, both in number and value, when measured against the control. The less formal approach seemed to appeal to our readers. I believe they felt it was more personal and therefore more valuable and trustworthy, even though it lacked the normal branding they are accustomed to.
Our opens for the unformatted one were nearly twice as many, as were click-throughs and donations and transactions were 50% higher. It was a dramatic result and we want to test out some of the micro details to see if it was a combination of things or one specific thing.