How personalizing a response on an instant donation page affects conversion Experiment ID: #21693

Hoover Institution

Experiment Summary

Ended On: 7/10/2020

Hoover Institution was promoting a content offer that was given away for free in exchange for the user’s name and email address. On the following page, the visitor was given the opportunity to make a gift. Since Hoover collected the first name of the user in the first step of the process, they wanted to test whether using that name to personalize the donation page affected results. They used dynamic keyword insertion, a feature of the landing page system, to insert the previously entered name into the first sentence on the page. Then, they launched a test to see how it affected results.

Research Question

Will personalizing a response on an instant donation page affect conversion?

Design

C: Control
T1: Treatment #1

Results

Treatment Name Conv. Rate Relative Difference Confidence
C: Control 1.7%
T1: Treatment #1 3.1% 83.4% 98.5%

This experiment has a required sample size of 902 in order to be valid. Since the experiment had a total sample size of 2,785, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 83.4% increase in conversion rate
× 0% increase in average gift

Key Learnings

The treatment, with the personalized “thank you” message, increased response by 83.4%. This implies that including someone’s name might bring them a few steps closer in the “relationship” required for the prospect to trust Hoover with a donation. However, due to the low overall number of gifts, this needs to be tested in other instances to see if the principle holds.


Experiment Documented by...

Jeff Giddens

Jeff is the President at NextAfter. If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.