Welcome to Josh Baker's Practical Advice for Optimizing Your Internet Marketing blog. Here you will find internet marketing optimization and online strategy articles full of tips, tricks, discussions, and thoughts to help you take your marketing and business to the next level of success.

When is Testing for Conversion Optimization not the Right Thing to Do?


There are just certain times when running a multivariate test to optimize web page conversions will produce unreliable results. Results that either will not yield statistically significant outcomes, or outcomes that even though the numbers may show statistical significance at the end of your test, would not be reliable enough to roll-out and see the nearly the same results much longer than after that particular testing period ends. Remember you are looking to take one step forward and improve your web pages conversion, and not two steps back rolling-out a page that ultimately performs worse than your control; which is quite possible if you are not mindful of certain instances.

Such instances include:

Seasonal traffic – Testing pages during specific high seasonal times for your business although may produce statistically significant outcomes by looking at the numbers themselves; the changes made based on the test outcomes would not be reliable after the seasonal traffic ends. The user intent during these times in most cases is not typical user intent or behavior displayed during the non-seasonal times, and in some cases also between seasonal times.

Traffic sources that fluctuate in delivery volume -If viewed at in a line chart would show high peaks and or low valleys (or may even show times of nonexistence traffic). This traffic volume is too unstable and therefore an unreliable indicator of ongoing performance. A specific instance would be running a multivariate test on a landing page that the traffic delivered to the test is from various different email campaigns. Also be careful of a test that suddenly receives a spike in traffic due to a current event for example that would send a large volume of traffic of non-typical visitors into your test.

Low traffic volumes – if your page does not receive enough consistent traffic of a certain volume than the likelihood of high confidence statistically significant results is slim-to-none in most cases. You need to have enough traffic to produce enough conversions (a conversion being anything you deem to be one, from a registration, to even a download) that your results will be accurate. Many conversion optimization experts say at least 10 conversions per day is the absolute minimum needed to run a test.

And if you’re A/B testing:

When you can’t run your control in unison with your test panels – without simultaneously running your control panel along with your test panels you will not be able to accurately assess the results of your test. You need to be able to assess how each of your panels or page combinations, both control and test panels, perform under the identical conditions and time period. The only way to accurately do so is to have them run simultaneously with your traffic randomly split amongst them.

Categories : Online Testing

Leave a Comment