Welcome to Josh Baker's Practical Advice for Optimizing Your Internet Marketing blog. Here you will find internet marketing optimization and online strategy articles full of tips, tricks, discussions, and thoughts to help you take your marketing and business to the next level of success.

Archive for Online Testing

There are just certain times when running a multivariate test to optimize web page conversions will produce unreliable results. Results that either will not yield statistically significant outcomes, or outcomes that even though the numbers may show statistical significance at the end of your test, would not be reliable enough to roll-out and see the nearly the same results much longer than after that particular testing period ends. Remember you are looking to take one step forward and improve your web pages conversion, and not two steps back rolling-out a page that ultimately performs worse than your control; which is quite possible if you are not mindful of certain instances.

Such instances include:

Seasonal traffic – Testing pages during specific high seasonal times for your business although may produce statistically significant outcomes by looking at the numbers themselves; the changes made based on the test outcomes would not be reliable after the seasonal traffic ends. The user intent during these times in most cases is not typical user intent or behavior displayed during the non-seasonal times, and in some cases also between seasonal times.

Traffic sources that fluctuate in delivery volume -If viewed at in a line chart would show high peaks and or low valleys (or may even show times of nonexistence traffic). This traffic volume is too unstable and therefore an unreliable indicator of ongoing performance. A specific instance would be running a multivariate test on a landing page that the traffic delivered to the test is from various different email campaigns. Also be careful of a test that suddenly receives a spike in traffic due to a current event for example that would send a large volume of traffic of non-typical visitors into your test.

Low traffic volumes – if your page does not receive enough consistent traffic of a certain volume than the likelihood of high confidence statistically significant results is slim-to-none in most cases. You need to have enough traffic to produce enough conversions (a conversion being anything you deem to be one, from a registration, to even a download) that your results will be accurate. Many conversion optimization experts say at least 10 conversions per day is the absolute minimum needed to run a test.

And if you’re A/B testing:

When you can’t run your control in unison with your test panels – without simultaneously running your control panel along with your test panels you will not be able to accurately assess the results of your test. You need to be able to assess how each of your panels or page combinations, both control and test panels, perform under the identical conditions and time period. The only way to accurately do so is to have them run simultaneously with your traffic randomly split amongst them.

Categories : Online Testing
Comments (0)

I have just a few ideas that I would like to see Google Website Optimizer (GWO) provide to its users (or at least some that I wish for).  I am big on documentation so it would be great to see some internal documentation and reporting features in the admin along with some more testing ability. By no means is this a complete list, and I should say that I am very happy with GWO – but these ideas popped into my head and thought I would share them for your comments or suggestions as well:

  • Date selection for each test – be able to select a time range for a test in order to look at the results by a user selected time period. This is more for curiosities sake on my part.


  •  Ability to add notes to tests – it would useful to be able to make notes about tests within the test admin itself for each test. Many times we have multiple people looking at a test and I would like to leave comments and get responses within the test, not only for ease but for permanent documentation. 
  • Allow the testing of more sections and areas – currently you can only test 8 sections and 127 variations. At least let me have 12 sections please!


  • Set up auto-emailing of daily reports – how nice it would be to get a daily or twice-daily email report of the current results that I could set up.


I haven’t thought this one through completely yet, not sure it’s recommendable – but the idea intrigues me:

  • Turn off test during certain time periods and auto turn back on – Here me out on this one first. Let’s say you are a lead generation model, you already turn your PPC campaigns off on weekends, etc. – all due to the fact that you have determined that weekend leads are ineffective, of poor quality and not-cost effective for you to follow-up with. Maybe you are a B2B and the weekend leads mainly consist of consumers. What if you could optimize for your weekday visitors only by having it shut off on Friday nights and turn back on Monday morning?


photo by Incessant Flux

Categories : Online Testing
Comments (0)

Three of my conversion optimization colleagues and I had a discussion online the other day that I had proposed about the common “sins” of online conversion testing we see or hear about often in organizations.  We came up with about 20 commons “sins” in about 7 minutes that we all agreed upon, and about 40 overall. Below you will find 8 of them in no particular order (with more to come in the future).

8 Common Sins of Online Conversion Testing that Organizations Let Happen:

  1. When running a multivariate test, after the test ends, not performing a head-to-head testing of the winning page combination and the control. The winning page combination is typically based on a prediction; a head-to-head test will further uncover the true results.

  3. Having too many people involved in the testing process AFTER the test is given the “go ahead”. Everyone involved should have a purpose otherwise the process slows down.

  5. Not believing that having no panels perform better than the control is still a win – just of a different kind; but only if you actually extract the knowledge hidden in your “loss”.

  7. Not setting a concrete conversion goal – know what your test hypothesis is and understand how you will analyze the data ahead of time. Alternate lessons may be and should be learned from a test but it’s vital to know exactly what and why you are testing something in the first place.

  9. Not allowing a test to run long-enough to accumulate enough conversions.

  11. Not running the control panel (this happens often) at the same exact time as the test panels.

  13. Letting personal opinions or biases override data in the results – the reason you test is because you really don’t know what will persuade your actual visitors best.

  15. No Patience – ending tests too early, or not allowing the process to happen as it should.


As bad as these are, we all agreed we were still happy that organizations have the desire to test!

Have an online conversion testing or optimization sin that you want to share or get off your chest? Let me know in the comments section.


You’ve already been running numerous tests on your best landing pages – those that contribute the highest value to your business. Unfortunately, sometimes you’ve run out of optimization ideas or hit a few roadblocks on what you should test next for even more conversion gains.  What should you do? Luckily, just as often when you are running a multivariate test or a/b test to improve the desired results of a given page on your website you will discover that you will gain improved conversion results not by altering a page element or adding a new or section to the page, but instead by removing one or more of your existing elements or sections.

Why is this so? Although each page, situation, and context is many times unique, a few of the more common reasons for the improvement in conversions include:

1) Removing distractions that enable the visitor to more clearly focus on your desired page goal.

2) Reducing the friction that forces the visitor to contemplate if the desired action is worth what is being asked of them to give in return.

3) Replacing confusing elements that prevent the visitor from understanding if they are on the correct page or even knowing what they are supposed to do next.

A few broader ranged ideas to consider include:

  • Removing to clear up page real estate
  • Removal to speed up page load time
  • Removal of potential road blocks or barriers

More detailed removal considerations include:

  • Removal of parts/all of navigation
  • Removal of sections of copy
  • Removal of unnecessary graphics
  • Removal of just the large file size images
  • Removal of flash elements (or those that require plug-ins or longer load time)
  • Removal of non-vital third party java-scripts
  • Removal of non-essential registration form fields
  • Removal of traffic-leaks
  • Removal of premiums or special offers

These should be enough start ideas to get you thinking in the right direction when you are looking at your landing page. You undoubtedly will develop various unique hypotheses for doing these (or any other “removal” ideas) based upon your own site’s data you have extracted and analyzed-or even from basic usability knowledge. The end goal is ultimately almost always the same – to uncover what page elements are negatively impacting your page’s ability to do its job properly so that you can fix them to increase the level of success your site achieves. Remember, removal testing doesn’t have to be done in isolation; removal can always be a part of any test when it’s appropriate to do so as judged by you.


If you are new to online testing and not sure what page or area to test on your website or just need that kick-start to get those testing adrenaline rushes back…

Here are 3 important areas to start pulling data for to get you going (or going again) on the forward path to optimization success.

1. The most visited pages on your website. Things to think about for each page – what’s the pages purpose, what’s the conversion rate, what’s the bounce rate, where are the leaks, what’s the average time spent on the page by your visitors, any coding errors hindering performance, page load time, special plug-ins needed for visitors to get full functionality.

2. Your Conversion points – Pull conversion data for each of your sites conversion points, how much revenue does each conversion point contribute, order each conversion point by revenue from producing the most to the least and look at the opportunities starting at the top of the list – a 100% increase in conversions on a page that only produces $50 won’t produce the same result as a 5% increase on a page that produces $10,000 in revenue – it’s a good place to start.

3. Your most popular visitor paths – Review data for your most popular visitor paths. Where are the leaks that visitors are exiting or straying from your desired end goal that you have designed for them?  What are the opportunities to optimize and keep your visitors on the desired path? Can you shorten the path if need be, work on your call-to-actions, add a newsletter signup box, and so on.

4. Bonus – Combinations of the above, i.e the most popular visited page with a conversion point, sorted by lowest conversion percentage with theoretical greatest chance for improvement.

Of course this is not the be all end all of what to look for or what to test in each area, but merely a good  refresher for those who need it, or a guiding hand for those confused with all the potential places to start testing first. But remember, it’s important to consider the opportunity costs in testing one area, page, path, etc. versus testing another.