Archive for AB testing
Simpson’s Paradox and Marketing Testing
Posted by: | CommentsIn 1973, the University of California at Berkeley was sued for showing bias in admissions for women to their graduate school. Men had a much better chance to be admitted than women according to the statistics given. The reporting showed that this sex bias was unlikely due to chance since the percentage difference between the men and women admitted was so large that it had to be in fact true.
But when the numbers were looked at by individual department, it was actually shown that there was a small but statistically significant bias that favored the women in actually having a higher chance at being admitted.
How can this be? Simple, it’s called Simpson’s Paradox. Simpson’s Paradox is when the trends derived from the data from individual subgroups are reversed when the groups are combined.
Google is A/B Testing Their Business Services Page
Posted by: | CommentsThis week I noticed Google has been A/B testing their Business Services page. I always love to see tests that other companies are doing.
Google actually has 3 test pages going up against their control. The major problem I see as a current user of the existing page to log into my accounts is that there is no link to Google Analytics from the test pages (yes I know I could go direct, but I go this route out of habit). But as we all know, the proof is in the data of what works best for the goals they are going after.
Click on the thumbnails to see the full size pages and let me know your thoughts.
Read More→
Testing Video Engagement Using Google Website Optimizer
Posted by: | CommentsUsing Minimum Viewing Time as a A/B Test Conversion Goal
Sample Scenario:
- You want to add a page with a video or demo to your web site.
- You believe that it is important for visitors to view as much as possible of the video.
- You have 2 videos to test against each other, and your test goal is to determine which one of your videos keeps more visitors watching (engaged) for at least a certain period of time and plan to keep the one that engages viewers more than the other.
In this scenario, since your test goal is to determine which video users are more engaged in watching for at least a certain period of time you will therefore want a conversion to trigger and to be recorded in GWO after that set period of predetermined minimum viewing time.
Back to the Scenario:
Read More→
How to Interpret the Estimated Conversion Rate Range in Google Website Optimizer
Posted by: | CommentsInterpreting the Estimated Conversion Rate Range Properly in GWO is of Key Importance!
It’s easy to get initially excited when you see that one of your test panels in Google Website Optimizer has a higher Estimated Conversion Rate than that of your control panel as presented on the Combination Report page. This may even lead to you believe you can end the test (prematurely I may add).
Unfortunately, just looking at the Conversion Rate number given to you by Google in the bold type font isn’t enough, you most certainly need to do a little bit of visualization to really have a better understanding of what is going on and how they are performing against each other.
Whether you are running an A/B test or a multivariate test, this is important to know for either – the number they give you is a conversion RATE RANGE. Many people mistakenly look at just the number given and do not visualize the full conversion range given along with it (done so with simple addition and subtraction of the number given next to the estimated conversion rate after the plus and minus sign). This range is based on the observed conversion rate of during the experiment thus far. Not factoring this in can lead to many people ending or wanting to end tests before they are truly ready to be ended. For example,
Estimated Conversion Rate
- Test Panel – 6.0% +/- 1.0%
- Control Panel – 5.5% +/- 1.0%
Reading and interpreting this correctly would actually tell you that the:
- Test Panel is converting in the range of 5.0% to 7.0%, and the
- Control Panel is converting in the range of 4.5% to 6.5%
This being true, their conversion rate ranges are overlapping each other. Visualizing this information shows you the overlap much more clearly as shown below:
Read More→
8 Common Sins of Online Conversion Testing that Organizations Let Happen
Posted by: | CommentsThree of my conversion optimization colleagues and I had a discussion online the other day that I had proposed about the common “sins” of online conversion testing we see or hear about often in organizations. We came up with about 20 commons “sins” in about 7 minutes that we all agreed upon, and about 40 overall. Below you will find 8 of them in no particular order (with more to come in the future).
8 Common Sins of Online Conversion Testing that Organizations Let Happen:
- When running a multivariate test, after the test ends, not performing a head-to-head testing of the winning page combination and the control. The winning page combination is typically based on a prediction; a head-to-head test will further uncover the true results.
- Having too many people involved in the testing process AFTER the test is given the “go ahead”. Everyone involved should have a purpose otherwise the process slows down.
- Not believing that having no panels perform better than the control is still a win – just of a different kind; but only if you actually extract the knowledge hidden in your “loss”.
- Not setting a concrete conversion goal – know what your test hypothesis is and understand how you will analyze the data ahead of time. Alternate lessons may be and should be learned from a test but it’s vital to know exactly what and why you are testing something in the first place.
- Not allowing a test to run long-enough to accumulate enough conversions.
- Not running the control panel (this happens often) at the same exact time as the test panels.
- Letting personal opinions or biases override data in the results – the reason you test is because you really don’t know what will persuade your actual visitors best.
- No Patience - ending tests too early, or not allowing the process to happen as it should.
As bad as these are, we all agreed we were still happy that organizations have the desire to test!
Have an online conversion testing or optimization sin that you want to share or get off your chest? Let me know in the comments section.
