Archive for Online Testing
Top 15 Tips For Multivariate Testing with Google Website Optimizer
Posted by: | CommentsI just got back from SIPA’s “Gain. Market. Share.” Conference at Planet Hollywood in Las Vegas yesterday. I along with Bob Brady and Kiplinger’s Greg Krehbiel hosted a session on Multivariate Testing with Google Website Optimizer. You can find our combined Top 15 To-Do List for Multivariate Testing with Google Website Optimizer that we concluded our session with over at the Mequoda Daily blog.
Three of my favorite sessions that I attended during the SIPA conference were those by Frank Gruber, Social Media Expert at AOL’s Social Media & Somewhat Obsession With Its Shiny Objects session; Michael Brito, Social Media Strategist at Intel Corporation’s Integrating Social Media into your Marketing session; and Bob Lorum, President of MarketingSherpa’s Maximize Your Landing Page Conversion Rates session.
MVT Vendor Detector Allows You To Spy on Your Competition’s Testing
Posted by: | CommentsAre you curious if your competition is performing any multivariate testing on their website? How about which vendor they are using and what areas of the page they are testing? Maybe you just want to know what your favorite site is doing as far as testing goes because you are fascinated with optimization just as I am.
Well now you can with the MVT Vendor Detector script for the Firefox Greasemonkey add-on. Both are free and install in seconds. Once installed, just go to any page and you will see a message in the top left-hand side of the screen if they are multivariate testing, what platform they are using, and the areas they are testing will be highlighted.
Here is a partial screenshot of it in action (click for larger version):
MVT vendor detector screenshot for multivariate testing insight
Who Are You Testing for When You Design Your Online Tests
Posted by: | CommentsIt’s much simpler to see large gains in conversions when you are just starting out with testing to improve your website, or even for those who are blindly performing tests on the basics of what already exists on their website such as button colors, headlines, and images to name just a few examples. These gains happen mostly because many companies place these random elements on their webpage’s because they thought they looked good, represented what they thought they should have on their page, or were possibly even told to do so. Simple testing of these elements for the first time often sees some gains because these pages were never tested for optimal performance in the first place when they were launched.
Who really is your customer? Do you really know your customer or do you just think you do.
Some of the biggest gains and major marketing-optimization-wins appear when you optimize with the customer (by customer I also mean potential customer) in mind. Uncovering who they truly are, why they arrived at your website and what they truly desire, and what can be done to make their experience towards your combined win-win goals better. Typically, you won’t have just one consistent answer to these questions, but instead multiple customer profiles, i.e. if you’re a company like TigerDirect where you may have various profiles and then variations on each such as the IT professional, the gadget-geek, the family looking for a computer, and others.
Your goal isn’t to convert more visitors, but instead your goal should be along the lines of optimizing your website to make it less frustrating, a clearer and more defined path, a more appealing presentation of your offer, more educating, less cluttered, or more engaging for your customers- all of which result in more conversions when you perform testing for the right combination. Knowing this enables you to understand what you are testing and how to continue past just changing button colors because you haven’t tried that color yet.
Uncover the necessary information about your customers and develop the customer profiles will help you in researching elements both currently known and not known (what you already have, and what you don’t have) to test on your website.
Optimizing the Cart Before the Horse?
Posted by: | CommentsWhen you are launching your website optimization program I can’t blame you for being anxious to jump in and start seeing those high double-digit conversion increases that is proudly displayed in almost every optimization article that you read. It makes me excited too just thinking about it. But wait; don’t put the optimization cart before the testing horse just yet. And if you are already in the midst of your optimization regimen rethinking the basics of site functionality, usability, and accessibility might just help you take your program to the next level.
Make sure that you site is working properly, if not, fix it first.
- If you are optimizing your checkout page and your server is consistently slow as a turtle, fix it.
- If your site times out every other time when the visitor hits the submit button, fix it.
- If the privacy policy link on your newsletter signup page goes to a 404 error, fix it .
These are just a few examples to get your brain moving in the right direction, but you get the point. These are no-brainer optimization “techniques” that need to be done first and will be likely to help you in your quest to achieve those double digit conversion results. What’s the point of optimizing a page or path consisting of errors unless your goal is a fully, and I use the term sarcastically, optimized, non-properly functioning page. This thought probably sounds like it should be followed by a “duh”, but trust me, go through the process a few times of checking your site, have a few others do it too – just to be that extra confident before launching that next test. A few simple fixes can be an automatic gain towards you ultimate goal.
Want to learn about the Hierarchy of Optimization? View the video Hierarchy of Optimization by Bryan Eisenberg of FutureNow and Scott Milrad of MarketMotive discussing Bryan’s theory or watch it below
Multivariate Testing; Fractional-Factorial, Full-Factorial, Taguchi – Huh? Part 1
Posted by: | CommentsWithin the past year there has been numerous published articles, blog posts, discussions, reports, and I am sure you could even find videos somewhere on the debate between fractional-factorial and full-factorial multivariate testing. Everything from one should only use full-factorial multivariate testing as fractional-factorial is not truly a statistically valid test type, to you should learn to incorporate both into your testing strategies, to everything-else in between.
So what’s the difference between fractional-factorial and full-factorial multivariate testing?
Full-Factorial multivariate testing actually tests all combinations of the options in your test. It takes longer than fractional-factorial since it does go through all of the possible combinations to deliver the results. For instance, if you have 3 page elements with 3 options each, you would have 27 (3x3x3) combinations. If you had 4 page elements, 2 with 3 options, and 2 with 2 options you would have 36 (3x3x2x2) combinations.
Fractional-Factorial multivariate testing does not test all of the combinations of options in your test but instead tests a smaller sampling of them thus allowing the test to run a shorter length to get your results. Most fractional-factorial testing programs then use math to determine these results based on this small sample. A common method used in fractional-factorial testing is the Taguchi Method.
Tim Ash, landing page guru of SiteTuners fame, blogged about the differences in his post titled Taguchi Sucks for Landing Page Testing . He states that:
“The principal drawbacks of fractional factorial methods are:
- Very small test sizes
- Restrictive & inflexible test designs
- Less accurate estimation of individual variable contributions
- Drawing the wrong conclusions
- Inability to consider context and variable interactions”
Fractional-Factorial vs. Full-Factorial: An Ideological War?, an article posted to Omniture’s Industry Insights blog covered their views on these two types of testing styles and why they believe each has their place, specifically when one doesn’t have the necessary amount of time or traffic to run a full-factorial test so that you can get the overall effects compared to the implied effects of each option. You’ll also find a great post by Avinash Kaushik in the comments section.
In the past, I have used fractional-factorial multivariate testing and before rolling out with any test winners have always split tested the best performing page versus our control page. To date, the page of favored options that performed best in our fractional-factorial test has consistently beat our control in the aft split test.
Google’s free Website Optimizer performs full-factorial testing along with the ability to see fractional-factorial data allowing you the ability to choose which data you believe you should follow.
Have opinions or comments on the debate between fractional and full factorial multivariate testing? I would love to hear them!
