KCBI

How the length of a pledge impacts email conversion rate

KCBI

Experiment Summary

Ended On: 9/15/2020

In an effort to acquire email addresses, KCBI created a pledge/petition for people to sign that aligned people’s beliefs with a core value of theirs. The original pledge had ten statements of belief within it and we wondered if the number of beliefs statements would impact conversion.

Ultimately what we were trying to figure out is this, can a pledge have too many belief statements resulting in friction and therefore drop-offs?

We cut the number of belief statements from 10 to 5 for the treatment. We chose the 5 for the treatment based on what we thought were the most “consumable” for people – they weren’t lengthy, weighted, and seemed easiest to understand, read, and align with.

Research Question

Does the length of a pledge/petition have a difference on the conversion rate?

Design

C: Long Pledge
T1: Short Pledge

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: 70.6%
T1: Short Pledge 77.2% 9.3% 95.8%

This experiment has a required sample size of 346 in order to be valid. Since the experiment had a total sample size of 740, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 9.3% increase in conversion rate
× 0% increase in average gift

Key Learnings

What we ended up seeing was a 9.3% increase in conversion for the shorter pledge! Our biggest takeaway here was that less is sometimes more. The pledge with 5 belief statements converted at a higher rate than the one with 10. This could be based on a couple of things…
1. The length of the pledge
2. Simplicity of the statements of beliefThe next step would be to look at the language and wording of each statement and begin to test there.

Now that we have seen that less is more we can begin iterating on what words or phrasing should be used.


Experiment Documented by Allan Torres
Allan Torres is an Optimization Associate at NextAfter.

Question about experiment #9794

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.