Josh Baker - Practical Advice for Optimizing your Internet Marketing » Online Testing

Top 7 Online Testing Pitfalls

Josh Baker — Thu, 05 May 2011 10:23:44 +0000

I was going through an old notebook the other day and found this gem from an Online Testing webinar that featured guest Ronny Kohavi, GM of the Microsoft Experimentation Platform.

Top 7 Pitfalls of Online Testing and Marketing Experimentation:

Wrong success metric
Incorrect interval calculation
Using standard formulas for standard deviations
Combining data when percent to treatment varies
Not filtering out robots
Invalid or inadequate instrumentation
Insufficient experimental control

8 Common Sins of Online Conversion Testing that Organizations Let Happen Three of my conversion optimization colleagues and I had a...
Boost your Online Testing Results for Impact: 3 Areas to Review If you are new to online testing and not sure...
Who Are You Testing for When You Design Your Online Tests It’s much simpler to see large gains in conversions when...

Google Website Optimizer Security Issue

Josh Baker — Sat, 11 Dec 2010 01:34:50 +0000

Google Website Optimizer announced this week that there is a potential security issue with their Website Optimizer. The basics of the vulnerability is that it would allow an attacker to execute malicious code on your site using a Cross-Site Scripting (XSS) attack, but only if the website or browser had already been compromised by a separate attack. Google says that the probability of the attack is low but that you should take the necessary action to protect your site from it.

Google has fixed the bug and any new experiments created on or after December 3, 2010 are not vulnerable. But if you have any experiments running that were created before December 3^rd, or that are paused or stopped that were created before that date you will need to update the code.

They have given two options to update your code which can be found here in the Official Google Website Optimizer blog post titled Update Your Website Optimizer scripts to secure your site, but your choices are to either stop your experiments and create new ones, or update the code on your running experiments directly. Google states that creating a new experiment is the simpler way between the two options.

How to Interpret the Estimated Conversion Rate Range in Google Website Optimizer Interpreting the Estimated Conversion Rate Range Properly in GWO is...
Top 15 Tips For Multivariate Testing with Google Website Optimizer I just got back from SIPA’s “Gain. Market. Share.” Conference at...
Testing Video Engagement Using Google Website Optimizer Using Minimum Viewing Time as a A/B Test Conversion Goal...

Google is A/B Testing Their Business Services Page

Josh Baker — Fri, 06 Aug 2010 18:11:07 +0000

This week I noticed Google has been A/B testing their Business Services page. I always love to see tests that other companies are doing.

Google actually has 3 test pages going up against their control. The major problem I see as a current user of the existing page to log into my accounts is that there is no link to Google Analytics from the test pages (yes I know I could go direct, but I go this route out of habit). But as we all know, the proof is in the data of what works best for the goals they are going after.

Click on the thumbnails to see the full size pages and let me know your thoughts.

Google Business Services Control:

Google Business Services Test Page A:

Google Business Services Test Page B:

Google Business Services Page C:

Google Testing their Business Solutions Page This morning I went to log into my Google Analytics...
Landing Page Optimization: Subtraction is the New Addition You’ve already been running numerous tests on your best landing...
Interpreting Google Website Optimizer’s Page Sections Report Google Website Optimizer, the free multivariate and AB testing tool...

How to Interpret the Estimated Conversion Rate Range in Google Website Optimizer

Josh Baker — Sun, 30 Aug 2009 15:30:11 +0000

Interpreting the Estimated Conversion Rate Range Properly in GWO is of Key Importance!

It’s easy to get initially excited when you see that one of your test panels in Google Website Optimizer has a higher Estimated Conversion Rate than that of your control panel as presented on the Combination Report page. This may even lead to you believe you can end the test (prematurely I may add).

Unfortunately, just looking at the Conversion Rate number given to you by Google in the bold type font isn’t enough, you most certainly need to do a little bit of visualization to really have a better understanding of what is going on and how they are performing against each other.

Whether you are running an A/B test or a multivariate test, this is important to know for either – the number they give you is a conversion RATE RANGE. Many people mistakenly look at just the number given and do not visualize the full conversion range given along with it (done so with simple addition and subtraction of the number given next to the estimated conversion rate after the plus and minus sign). This range is based on the observed conversion rate of during the experiment thus far. Not factoring this in can lead to many people ending or wanting to end tests before they are truly ready to be ended. For example,

Estimated Conversion Rate

Test Panel – 6.0% +/- 1.0%
Control Panel – 5.5% +/- 1.0%

Reading and interpreting this correctly would actually tell you that the:

Test Panel is converting in the range of 5.0% to 7.0%, and the
Control Panel is converting in the range of 4.5% to 6.5%

This being true, their conversion rate ranges are overlapping each other. Visualizing this information shows you the overlap much more clearly as shown below:

So, is the test panel really converting better than the control? Potentially the control panel could still be converting better – it’s not clear enough to make a decisive or correct decision at this point.

Now think of a scenario such as two panels having a higher conversion rate and much larger conversion rate range:

Test – 55.0% +/- 8.0% (conversion rate range of 47.0% to 63.0%)
Control – 52.0% +/- 7.0% (conversion rate range of 45.0% to 59.0%)

Some may feel that the test panel is the winner and so it’s ok to stop the test and roll out with the test panel…BUT…

As the test goes on for a longer period of time, you should find the conversion rate range getting “tighter” and the above test potentially ending in such as a way as:

Control panel converting at 58% +/- 1 .0%, and the
Test panel converting at 47.0% +/- 1.0%.

In this ending scenario the control panel is actually the winner not the test panel as we saw mistakenly earlier on. Ending the test too early certainly changes the story drastically as does the results could differ as well!

I know sometimes it’s exciting to see your test panel winning, but patience is certainly of high importance in A/B and multivariate testing in order to make sure your tests and results are always moving forward correctly.

4 Items for My Google Website Optimizer Wishlist I have just a few ideas that I would like...
Top 15 Tips For Multivariate Testing with Google Website Optimizer I just got back from SIPA’s “Gain. Market. Share.” Conference at...
Interpreting Google Website Optimizer’s Page Sections Report Google Website Optimizer, the free multivariate and AB testing tool...

When is Testing for Conversion Optimization not the Right Thing to Do?

Josh Baker — Sat, 16 May 2009 20:52:18 +0000

There are just certain times when running a multivariate test to optimize web page conversions will produce unreliable results. Results that either will not yield statistically significant outcomes, or outcomes that even though the numbers may show statistical significance at the end of your test, would not be reliable enough to roll-out and see the nearly the same results much longer than after that particular testing period ends. Remember you are looking to take one step forward and improve your web pages conversion, and not two steps back rolling-out a page that ultimately performs worse than your control; which is quite possible if you are not mindful of certain instances.

Such instances include:

Seasonal traffic – Testing pages during specific high seasonal times for your business although may produce statistically significant outcomes by looking at the numbers themselves; the changes made based on the test outcomes would not be reliable after the seasonal traffic ends. The user intent during these times in most cases is not typical user intent or behavior displayed during the non-seasonal times, and in some cases also between seasonal times.

Traffic sources that fluctuate in delivery volume -If viewed at in a line chart would show high peaks and or low valleys (or may even show times of nonexistence traffic). This traffic volume is too unstable and therefore an unreliable indicator of ongoing performance. A specific instance would be running a multivariate test on a landing page that the traffic delivered to the test is from various different email campaigns. Also be careful of a test that suddenly receives a spike in traffic due to a current event for example that would send a large volume of traffic of non-typical visitors into your test.

Low traffic volumes – if your page does not receive enough consistent traffic of a certain volume than the likelihood of high confidence statistically significant results is slim-to-none in most cases. You need to have enough traffic to produce enough conversions (a conversion being anything you deem to be one, from a registration, to even a download) that your results will be accurate. Many conversion optimization experts say at least 10 conversions per day is the absolute minimum needed to run a test.

And if you’re A/B testing:

When you can’t run your control in unison with your test panels – without simultaneously running your control panel along with your test panels you will not be able to accurately assess the results of your test. You need to be able to assess how each of your panels or page combinations, both control and test panels, perform under the identical conditions and time period. The only way to accurately do so is to have them run simultaneously with your traffic randomly split amongst them.

8 Common Sins of Online Conversion Testing that Organizations Let Happen Three of my conversion optimization colleagues and I had a...
A/B Testing Low Traffic Web Pages Sometimes the website or the page you want to test...