NextAfter

How sending a webinar invite from the host affects email click rates

Experiment ID: #183954

NextAfter

Experiment Summary

Ended On: 06/25/2024

We all know and love our Audience Engagement Manager, Riley, and so do our subscribers. But for this test, we wanted to know what would happen if we sent a webinar invite from the webinar’s host (Nathan), rather than our usual sender, (Riley).

Our previous tests have shown that the more we can humanize our emails (I.e., make them look and feel as much like a personal 1:1 email as possible), the better our engagement rates.

So we wondered if sending the email from the host would make the email seem more like a personal invitation and less like a promotion — and how this would affect engagement rates.

For this test, we split the traffic 50/50 with roughly one-half of subscribers receiving the invite from Riley (with “we” language) and the other half receiving the invite from Nathan (with “I” language).

Research Question

We believe that sending a webinar invite from the host will increase click rates by humanizing the email and creating a feeling of liking.

Design

C: Control

T1: Treatment #1

Results

	Treatment Name	Click Rate	Relative Difference	Confidence
C:	Control	0.94%
T1:	Treatment #1	1.3%	36.1%	97.5%

This experiment has a required sample size of 7,299 in order to be valid. Since the experiment had a total sample size of 19,161, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

36.1% increase in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

Sending the email from the webinar's host produced a 31.7% statistically valid lift in email click rates, smashing the control. So this seems like a clear-cut learning -- by sending the email from the webinar's host, we humanized the interaction and earned a bunch more clicks, end of story, right? ... not so fast, my friend! As we dug deeper into the data, we discovered an unexpected incongruency in the click rates and registration rates between the control and the treatment. While the treatment (from the host) produced a significant 32% increase in clicks, it ultimately led to a 167% decrease in sign-up rates. This one's a head-scratcher. Considering that the changes to the treatment were tightly aligned with our hypothesis, and no other changes were made (either to the email or the following landing page), we would not expect to see the treatment produce such a large lift in click rates but such a gigantic decrease in sign-up rates. The only element we could isolate in our analysis is that the control email shares Nathan's title while the treatment shares it only in Nathan's signature, in gray font. Perhaps the demonstration of "authority" made a bigger impact in people's decision to register. Clearly, humanizing the email improved click rates, which is what we set out to test ... what's less clear is why this tactic caused such a dramatic dip in registration rates, warranting closer analysis and a second test.

Experiment Documented by NextAfter

Question about experiment #183954

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.