“Getting a negative test result is the worst thing that can happen when you perform an A/B test – right?”
While many marketers probably would subscribe to the logic behind this statement, experience from almost 400 split tests tells me otherwise. A negative test result will often give you as much insight and learning as a test that generated a major lift.
Here’s a great example of a test where I was very confident that my treatment would kick ass. In fact, I didn’t even perform the test to see whether the treatment would generate a lift, I performed the test to see how much of a lift it would generate.
The client here is Bettingexpert.com an international betting community. I was hired to optimize the home page with the goal of getting more potential users to sign up for a membership.
As mentioned, I felt very confident while setting up the test, and I was excited about the prospect of seeing how much better the treatment would perform in real life. Boy did I have another thing coming!
My tests indicated that the word spam had an undesirable effect – even when used to assure visitors that they would not receive any spam. My hypothesis is that by placing the word spam in close proximity of the form, you actually plant an idea in the minds of the prospects; “Oh wait, could they actually end up spamming me?”
One might be inclined to view some of the case studies in this article as bad tests. But in fact they weren’t bad tests at all because they provided important insights that eventually lead to serious conversion lifts on a number of different landing pages.
Of course hitting a home run in the first swing is easier on the ego. But when you approach optimization as a process – not a one-off opportunity to swing for the fences – you’ll see that stopping at a few bases along the way is often what it takes to win the game.