Illinois Policy Institute

How a subject line’s perceived impact on the subscriber influences click through rate

Experiment ID: #6326

Illinois Policy Institute

Experiment Summary

Timeframe: 03/07/2017 - 03/17/2017

In an effort to educate their subscriber base on the public’s sentiment toward the state’s current budget crisis, the Illinois Policy Institute sent an email with the results of an independent 3rd party poll. The results of the poll aligned with the message IPI had been promoting since the current session began. It also supported the budget plan the the organization had developed and was working with state legislators to pass.

The goal of the email was to educate and engage the subscriber file so we decided to test out two subject lines that highlighted the contents of the message: “tax hikes” and “overspending.”

Research Question

Which subject line will produce the greatest email engagement (opens and clicks)?

Design

C: tax hikes
T1: overspending

Results

 Treatment NameClick RateRelative DifferenceConfidence
C: tax hikes 0.81%
T1: overspending 0.62%-23.1% 99.3%

This experiment has a required sample size of 15,628 in order to be valid. Since the experiment had a total sample size of 60,031, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    23.1% decrease in traffic
× 0% increase in conversion rate
× 0% increase in average gift

Key Learnings

After sending to half of the email file, we found that “overspending” decreased the overall click through rate by 23%. It also negatively impacted the open rate by 5%. It is our hypothesis that this behavior is due to how the words are perceived by the subscriber. A “tax hike” is something that would impact them directly while “overspending” is something that they may not be in favor of but does not appear have any direct impact on them.

This serves as an important reminder that all aspects of the messaging should focus on how the topic will impact the recipient (when possible). We need to address not only why a message is important but why it is important to the reader.


Experiment Documented by NextAfter

Question about experiment #6326

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.