Archive for Online Testing
Top 7 Online Testing Pitfalls
Posted by: | CommentsI was going through an old notebook the other day and found this gem from an Online Testing webinar that featured guest Ronny Kohavi, GM of the Microsoft Experimentation Platform.
Top 7 Pitfalls of Online Testing and Marketing Experimentation:
- Wrong success metric
- Incorrect interval calculation
- Using standard formulas for standard deviations
- Combining data when percent to treatment varies
- Not filtering out robots
- Invalid or inadequate instrumentation
- Insufficient experimental control
Google Website Optimizer Security Issue
Posted by: | CommentsGoogle Website Optimizer announced this week that there is a potential security issue with their Website Optimizer. The basics of the vulnerability is that it would allow an attacker to execute malicious code on your site using a Cross-Site Scripting (XSS) attack, but only if the website or browser had already been compromised by a separate attack. Google says that the probability of the attack is low but that you should take the necessary action to protect your site from it.
Google has fixed the bug and any new experiments created on or after December 3, 2010 are not vulnerable. But if you have any experiments running that were created before December 3rd, or that are paused or stopped that were created before that date you will need to update the code.
They have given two options to update your code which can be found here in the Official Google Website Optimizer blog post titled Update Your Website Optimizer scripts to secure your site, but your choices are to either stop your experiments and create new ones, or update the code on your running experiments directly. Google states that creating a new experiment is the simpler way between the two options.
Google is A/B Testing Their Business Services Page
Posted by: | CommentsThis week I noticed Google has been A/B testing their Business Services page. I always love to see tests that other companies are doing.
Google actually has 3 test pages going up against their control. The major problem I see as a current user of the existing page to log into my accounts is that there is no link to Google Analytics from the test pages (yes I know I could go direct, but I go this route out of habit). But as we all know, the proof is in the data of what works best for the goals they are going after.
Click on the thumbnails to see the full size pages and let me know your thoughts.
Read More→
How to Interpret the Estimated Conversion Rate Range in Google Website Optimizer
Posted by: | CommentsInterpreting the Estimated Conversion Rate Range Properly in GWO is of Key Importance!
It’s easy to get initially excited when you see that one of your test panels in Google Website Optimizer has a higher Estimated Conversion Rate than that of your control panel as presented on the Combination Report page. This may even lead to you believe you can end the test (prematurely I may add).
Unfortunately, just looking at the Conversion Rate number given to you by Google in the bold type font isn’t enough, you most certainly need to do a little bit of visualization to really have a better understanding of what is going on and how they are performing against each other.
Whether you are running an A/B test or a multivariate test, this is important to know for either – the number they give you is a conversion RATE RANGE. Many people mistakenly look at just the number given and do not visualize the full conversion range given along with it (done so with simple addition and subtraction of the number given next to the estimated conversion rate after the plus and minus sign). This range is based on the observed conversion rate of during the experiment thus far. Not factoring this in can lead to many people ending or wanting to end tests before they are truly ready to be ended. For example,
Estimated Conversion Rate
- Test Panel – 6.0% +/- 1.0%
- Control Panel – 5.5% +/- 1.0%
Reading and interpreting this correctly would actually tell you that the:
- Test Panel is converting in the range of 5.0% to 7.0%, and the
- Control Panel is converting in the range of 4.5% to 6.5%
This being true, their conversion rate ranges are overlapping each other. Visualizing this information shows you the overlap much more clearly as shown below:
Read More→
When is Testing for Conversion Optimization not the Right Thing to Do?
Posted by: | CommentsThere are just certain times when running a multivariate test to optimize web page conversions will produce unreliable results. Results that either will not yield statistically significant outcomes, or outcomes that even though the numbers may show statistical significance at the end of your test, would not be reliable enough to roll-out and see the nearly the same results much longer than after that particular testing period ends. Remember you are looking to take one step forward and improve your web pages conversion, and not two steps back rolling-out a page that ultimately performs worse than your control; which is quite possible if you are not mindful of certain instances.
Such instances include:
Seasonal traffic – Testing pages during specific high seasonal times for your business although may produce statistically significant outcomes by looking at the numbers themselves; the changes made based on the test outcomes would not be reliable after the seasonal traffic ends. The user intent during these times in most cases is not typical user intent or behavior displayed during the non-seasonal times, and in some cases also between seasonal times.
Traffic sources that fluctuate in delivery volume -If viewed at in a line chart would show high peaks and or low valleys (or may even show times of nonexistence traffic). This traffic volume is too unstable and therefore an unreliable indicator of ongoing performance. A specific instance would be running a multivariate test on a landing page that the traffic delivered to the test is from various different email campaigns. Also be careful of a test that suddenly receives a spike in traffic due to a current event for example that would send a large volume of traffic of non-typical visitors into your test.
Low traffic volumes – if your page does not receive enough consistent traffic of a certain volume than the likelihood of high confidence statistically significant results is slim-to-none in most cases. You need to have enough traffic to produce enough conversions (a conversion being anything you deem to be one, from a registration, to even a download) that your results will be accurate. Many conversion optimization experts say at least 10 conversions per day is the absolute minimum needed to run a test.
And if you’re A/B testing:
When you can’t run your control in unison with your test panels – without simultaneously running your control panel along with your test panels you will not be able to accurately assess the results of your test. You need to be able to assess how each of your panels or page combinations, both control and test panels, perform under the identical conditions and time period. The only way to accurately do so is to have them run simultaneously with your traffic randomly split amongst them.
