Welcome

Welcome to Josh Baker's Practical Advice for Optimizing Your Internet Marketing blog. Here you will find internet marketing optimization and online strategy articles full of tips, tricks, discussions, and thoughts to help you take your marketing and business to the next level of success.
Oct
20

Multivariate Testing; Fractional-Factorial, Full-Factorial, Taguchi – Huh? Part 1

By

Within the past year there has been numerous published articles, blog posts, discussions, reports, and I am sure you could even find videos somewhere  on the debate between fractional-factorial and full-factorial multivariate testing. Everything from one should only use full-factorial multivariate testing as fractional-factorial is not truly a statistically valid test type, to you should learn to incorporate both into your testing strategies, to everything-else in between.

So what’s the difference between fractional-factorial and full-factorial multivariate testing?

Full-Factorial multivariate testing actually tests all combinations of the options in your test. It takes longer than fractional-factorial since it does go through all of the possible combinations to deliver the results. For instance, if you have 3 page elements with 3 options each, you would have 27 (3x3x3) combinations. If you had 4 page elements, 2 with 3 options, and 2 with 2 options you would have 36 (3x3x2x2) combinations.

Fractional-Factorial multivariate testing does not test all of the combinations of options in your test but instead tests a smaller sampling of them thus allowing the test to run a shorter length to get your results. Most fractional-factorial testing programs then use math to determine these results based on this small sample. A common method used in fractional-factorial testing is the Taguchi Method.

Tim Ash, landing page guru of SiteTuners fame, blogged about the differences in his post titled Taguchi Sucks for Landing Page Testing . He states that:

“The principal drawbacks of fractional factorial methods are:

  • Very small test sizes
  • Restrictive & inflexible test designs
  • Less accurate estimation of individual variable contributions
  • Drawing the wrong conclusions
  • Inability to consider context and variable interactions”


Fractional-Factorial vs. Full-Factorial: An Ideological War?
, an article posted to Omniture’s Industry Insights blog covered their views on these two types of testing styles and why they believe each has their place, specifically when one doesn’t have the necessary amount of time or traffic to run a full-factorial test so that you can get the overall effects compared to the implied effects of each option.¬† You’ll also find a great post by Avinash Kaushik in the comments section.

In the past, I have used fractional-factorial multivariate testing and before rolling out with any test winners have always split tested the best performing page versus our control page.  To date, the page of favored options that performed best in our fractional-factorial test has consistently beat our control in the aft split test.

Google’s free Website Optimizer performs full-factorial testing along with the ability to see fractional-factorial data allowing you the ability to choose which data you believe you should follow.

Have opinions or comments on the debate between fractional and full factorial multivariate testing? I would love to hear them!

Share

2 Comments

1

It’s nice to hear that you’ve had success with fractional-factorial multivariate testing! I’m glad you found our blog post helpful and I’m looking forward to your next installment.

– Lily at Omniture

2

You should also look at ADO ( accenture tool) best for full factorial and uses different modelling requiring hardly nay sample size.

Leave a Comment