Welcome to Josh Baker's Practical Advice for Optimizing Your Internet Marketing blog. Here you will find internet marketing optimization and online strategy articles full of tips, tricks, discussions, and thoughts to help you take your marketing and business to the next level of success.

8 Common Sins of Online Conversion Testing that Organizations Let Happen


Three of my conversion optimization colleagues and I had a discussion online the other day that I had proposed about the common “sins” of online conversion testing we see or hear about often in organizations.  We came up with about 20 commons “sins” in about 7 minutes that we all agreed upon, and about 40 overall. Below you will find 8 of them in no particular order (with more to come in the future).

8 Common Sins of Online Conversion Testing that Organizations Let Happen:

  1. When running a multivariate test, after the test ends, not performing a head-to-head testing of the winning page combination and the control. The winning page combination is typically based on a prediction; a head-to-head test will further uncover the true results.

  3. Having too many people involved in the testing process AFTER the test is given the “go ahead”. Everyone involved should have a purpose otherwise the process slows down.

  5. Not believing that having no panels perform better than the control is still a win – just of a different kind; but only if you actually extract the knowledge hidden in your “loss”.

  7. Not setting a concrete conversion goal – know what your test hypothesis is and understand how you will analyze the data ahead of time. Alternate lessons may be and should be learned from a test but it’s vital to know exactly what and why you are testing something in the first place.

  9. Not allowing a test to run long-enough to accumulate enough conversions.

  11. Not running the control panel (this happens often) at the same exact time as the test panels.

  13. Letting personal opinions or biases override data in the results – the reason you test is because you really don’t know what will persuade your actual visitors best.

  15. No Patience – ending tests too early, or not allowing the process to happen as it should.


As bad as these are, we all agreed we were still happy that organizations have the desire to test!

Have an online conversion testing or optimization sin that you want to share or get off your chest? Let me know in the comments section.




#3 is a big one for me. Took a while to accept the fact that failures were basically successes, because if it weren’t for testing, I would have just guessed and caused unknown negative impact.

Great post Josh.


Great post. As a landing page optimization consultant I completely agree with your 8 sins especially with #6 about not running the control panel at the same time as the test panels. Often businesses that hire me tell me about why they implemented some changes and the tests are often not against the control but the control first, then the test page after and they compare results.

I’d like to add a #9 – Not all tests will have high conversion “winners” everytime.


Great list. I’ve seen all of those! Another of my favorites is “Testing two wildly different things against each other and then assuming that every element of the one that performs better is actually better.” Sometimes you need to test very small changes against each other just to be able to narrow down what’s really making the difference. After all, if you change the text, the layout, AND the call to action, can you really say which of those is making the biggest difference in a multivariate test?

I still LOVE multivariate testing, but it is so easy to go wrong without experience and guidance.

Thanks for the post!


Social comments and analytics for this post…

This post was mentioned on Twitter by RexDixon: 8 Common Sins of Online Conversion Testing that Organizations Let Happen – http://bit.ly/9jQnOx #conversions #measure #abtests /via @abtests…

Leave a Comment