Will a shorter, more direct email lead to more clicks?
NextAfter
Experiment Summary
Ended On: 08/17/2020
NextAfter’s annual conference — the Nonprofit Innovation & Optimization Summit — moved to a virtual event due to COVID-19 but the change also further delayed the launch and promotion. For the first launch email, we wondered if a shorter, more to the point email may actually do better in driving clicks, engagement, and registrations (free and paid) as a lot of people on our list we (assume) know the event, know it’s quality, and may even be expecting the announcement so we didn’t need to spend as much time in the email explaining the event and sharing its value.
Research Question
Will as shorter, more direct email lead to more clicks?
Design
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 3.6% | ||
| T1: | Treatment #1 | 5.0% | 39.9% | 100.0% |
This experiment has a required sample size of 1,539 in order to be valid. Since the experiment had a total sample size of 20,584, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
39.9% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift
Key Learnings
The shorter, more direct email significantly increased the number of clicks and people who went on to visit the landing/registration page. The goal of the email, of course, wasn’t just to get clicks but to get registrations (both free and paid) so we wanted to look at that as well and here’s what we found for free and paid registrations:
- The treatment (shorter version) helped increase free registrations 46.6% (99.9% LoC – experiment here)
- The treatment (shorter version) saw a decrease in paid registrations of 60% (74.3% LoC – experiment here)
The shorter email not only sent much more traffic to the registration page but those that visited the page converted at almost the exact same rate (31.7% vs. 32.7%) which seems to validate that those who were interested in attending with a free ticket didn’t need the extra copy at all to help convince them to register. We did see a lift, but not significant, for paid ticket registrations with the longer copy and a directional lift in paid registration conversion rate for those who clicked from the longer version so there may be a relationship with longer copy and paid registration but in this case it both wasn’t clear and not worth the extra copy/time for overall performance.