Top 7 Online Testing Pitfalls

I was going through an old notebook the other day and found this gem from an Online Testing webinar that featured guest Ronny Kohavi, GM of the Microsoft Experimentation Platform.

Top 7 Pitfalls of Online Testing and Marketing Experimentation:

1. Wrong success metric
2. Incorrect interval calculation
3. Using standard formulas for standard deviations
4. Combining data when percent to treatment varies
5. Not filtering out robots
6. Invalid or inadequate instrumentation
7. Insufficient experimental control

Related posts:

1. 8 Common Sins of Online Conversion Testing that Organizations Let Happen Three of my conversion optimization colleagues and I had a...
2. Boost your Online Testing Results for Impact: 3 Areas to Review If you are new to online testing and not sure...
3. Who Are You Testing for When You Design Your Online Tests It’s much simpler to see large gains in conversions when...
4. When is Testing for Conversion Optimization not the Right Thing to Do? There are just certain times when running a multivariate test...
5. AB Testing or Multivariate Testing – The Laymans Explanation In most instances of classic A/B testing, you alter only...