“Getting a negative test result is the worst thing that can happen when you perform an A/B test – right?”

While many marketers probably would subscribe to the logic behind this statement, experience from almost 400 split tests tells me otherwise. A negative test result will often give you as much insight and learning as a test that generated a major lift.

Here’s a great example of a test where I was very confident that my treatment would kick ass. In fact, I didn’t even perform the test to see whether the treatment would generate a lift, I performed the test to see how much of a lift it would generate.

The client here is an international betting community. I was hired to optimize the home page with the goal of getting more potential users to sign up for a membership.

As mentioned, I felt very confident while setting up the test, and I was excited about the prospect of seeing how much better the treatment would perform in real life. Boy did I have another thing coming!

The treatment with the privacy policy reduced conversions by 18.70%. Again I was humbled and taken aback by the test data. The result was completely counter-intuitive but it taught me an important lesson.

This negative test result really got my mind going, and I decided to run a number of follow-up experiments to learn more about what made this particular privacy policy have a negative impact in the mind of the prospects.

My tests indicated that the word spam had an undesirable effect – even when used to assure visitors that they would not receive any spam. My hypothesis is that by placing the word spam in close proximity of the form, you actually plant an idea in the minds of the prospects; “Oh wait, could they actually end up spamming me?”

I’ve performed privacy policy tests on other landing pages, and the results have been quite similar. Here’s en example from the landing page for my free CRO e-book 7 Universal Conversion Optimization Principles where I saw a 24% drop in downloads when ran the original “I guarantee 100% privacy.” Against “100% privacy – I will never spam you!”

One might be inclined to view some of the case studies in this article as bad tests. But in fact they weren’t bad tests at all because they provided important insights that eventually lead to serious conversion lifts on a number of different landing pages.

Of course hitting a home run in the first swing is easier on the ego. But when you approach optimization as a process – not a one-off opportunity to swing for the fences – you’ll see that stopping at a few bases along the way is often what it takes to win the game.


18.7% decrease in signups

Simply adding a privacy policy doesn’t guarantee more sing-ups, in fact it can seriously reduce your conversion rate.


Replacing McAfee badges with TrustedSite badges

Leave a Reply

Your email address will not be published. Required fields are marked *