Will a shorter, more direct email lead to more clicks? | NextAfter
NextAfter

Will a shorter, more direct email lead to more clicks?

Experiment ID: #21354

NextAfter

Experiment Summary

Ended On: 8/17/2020

NextAfter’s annual conference — the Nonprofit Innovation & Optimization Summit — moved to a virtual event due to COVID-19 but the change also further delayed the launch and promotion. For the first launch email, we wondered if a shorter, more to the point email may actually do better in driving clicks, engagement, and registrations (free and paid) as a lot of people on our list we (assume) know the event, know it’s quality, and may even be expecting the announcement so we didn’t need to spend as much time in the email explaining the event and sharing its value.

Research Question

Will as shorter, more direct email lead to more clicks?

Results

Treatment Name Click Rate Relative Difference Confidence
C: Control 3.6%
T1: Treatment #1 5.0% 39.9% 100.0%

This experiment has a required sample size of 1,539 in order to be valid. Since the experiment had a total sample size of 20,584, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    39.9% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The shorter, more direct email significantly increased the number of clicks and people who went on to visit the landing/registration page. The goal of the email, of course, wasn’t just to get clicks but to get registrations (both free and paid) so we wanted to look at that as well and here’s what we found for free and paid registrations:

  • The treatment (shorter version) helped increase free registrations 46.6% (99.9% LoC – experiment here)
  • The treatment (shorter version) saw a decrease in paid registrations of 60% (74.3% LoC – experiment here)

The shorter email not only sent much more traffic to the registration page but those that visited the page converted at almost the exact same rate (31.7% vs. 32.7%) which seems to validate that those who were interested in attending with a free ticket didn’t need the extra copy at all to help convince them to register. We did see a lift, but not significant, for paid ticket registrations with the longer copy and a directional lift in paid registration conversion rate for those who clicked from the longer version so there may be a relationship with longer copy and paid registration but in this case it both wasn’t clear and not worth the e


Experiment Documented by Kevin Peters

Kevin is the Chief Technology Officer at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.

Question about experiment #6637

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.