Welcome

Welcome to Josh Baker's Practical Advice for Optimizing Your Internet Marketing blog. Here you will find internet marketing optimization and online strategy articles full of tips, tricks, discussions, and thoughts to help you take your marketing and business to the next level of success.

Archive for AB testing

Three of my conversion optimization colleagues and I had a discussion online the other day that I had proposed about the common “sins” of online conversion testing we see or hear about often in organizations.  We came up with about 20 commons “sins” in about 7 minutes that we all agreed upon, and about 40 overall. Below you will find 8 of them in no particular order (with more to come in the future).

8 Common Sins of Online Conversion Testing that Organizations Let Happen:

  1. When running a multivariate test, after the test ends, not performing a head-to-head testing of the winning page combination and the control. The winning page combination is typically based on a prediction; a head-to-head test will further uncover the true results.
  2.  

  3. Having too many people involved in the testing process AFTER the test is given the “go ahead”. Everyone involved should have a purpose otherwise the process slows down.
  4.  

  5. Not believing that having no panels perform better than the control is still a win – just of a different kind; but only if you actually extract the knowledge hidden in your “loss”.
  6.  

  7. Not setting a concrete conversion goal – know what your test hypothesis is and understand how you will analyze the data ahead of time. Alternate lessons may be and should be learned from a test but it’s vital to know exactly what and why you are testing something in the first place.
  8.  

  9. Not allowing a test to run long-enough to accumulate enough conversions.
  10.  

  11. Not running the control panel (this happens often) at the same exact time as the test panels.
  12.  

  13. Letting personal opinions or biases override data in the results – the reason you test is because you really don’t know what will persuade your actual visitors best.
  14.  

  15. No Patience – ending tests too early, or not allowing the process to happen as it should.

 

As bad as these are, we all agreed we were still happy that organizations have the desire to test!

Have an online conversion testing or optimization sin that you want to share or get off your chest? Let me know in the comments section.

Share

You’ve already been running numerous tests on your best landing pages – those that contribute the highest value to your business. Unfortunately, sometimes you’ve run out of optimization ideas or hit a few roadblocks on what you should test next for even more conversion gains.  What should you do? Luckily, just as often when you are running a multivariate test or a/b test to improve the desired results of a given page on your website you will discover that you will gain improved conversion results not by altering a page element or adding a new or section to the page, but instead by removing one or more of your existing elements or sections.

Why is this so? Although each page, situation, and context is many times unique, a few of the more common reasons for the improvement in conversions include:

1) Removing distractions that enable the visitor to more clearly focus on your desired page goal.

2) Reducing the friction that forces the visitor to contemplate if the desired action is worth what is being asked of them to give in return.

3) Replacing confusing elements that prevent the visitor from understanding if they are on the correct page or even knowing what they are supposed to do next.

A few broader ranged ideas to consider include:

  • Removing to clear up page real estate
  • Removal to speed up page load time
  • Removal of potential road blocks or barriers

More detailed removal considerations include:

  • Removal of parts/all of navigation
  • Removal of sections of copy
  • Removal of unnecessary graphics
  • Removal of just the large file size images
  • Removal of flash elements (or those that require plug-ins or longer load time)
  • Removal of non-vital third party java-scripts
  • Removal of non-essential registration form fields
  • Removal of traffic-leaks
  • Removal of premiums or special offers

These should be enough start ideas to get you thinking in the right direction when you are looking at your landing page. You undoubtedly will develop various unique hypotheses for doing these (or any other “removal” ideas) based upon your own site’s data you have extracted and analyzed-or even from basic usability knowledge. The end goal is ultimately almost always the same – to uncover what page elements are negatively impacting your page’s ability to do its job properly so that you can fix them to increase the level of success your site achieves. Remember, removal testing doesn’t have to be done in isolation; removal can always be a part of any test when it’s appropriate to do so as judged by you.

Share

If you are new to online testing and not sure what page or area to test on your website or just need that kick-start to get those testing adrenaline rushes back…

Here are 3 important areas to start pulling data for to get you going (or going again) on the forward path to optimization success.

1. The most visited pages on your website. Things to think about for each page – what’s the pages purpose, what’s the conversion rate, what’s the bounce rate, where are the leaks, what’s the average time spent on the page by your visitors, any coding errors hindering performance, page load time, special plug-ins needed for visitors to get full functionality.

2. Your Conversion points – Pull conversion data for each of your sites conversion points, how much revenue does each conversion point contribute, order each conversion point by revenue from producing the most to the least and look at the opportunities starting at the top of the list – a 100% increase in conversions on a page that only produces $50 won’t produce the same result as a 5% increase on a page that produces $10,000 in revenue – it’s a good place to start.

3. Your most popular visitor paths – Review data for your most popular visitor paths. Where are the leaks that visitors are exiting or straying from your desired end goal that you have designed for them?  What are the opportunities to optimize and keep your visitors on the desired path? Can you shorten the path if need be, work on your call-to-actions, add a newsletter signup box, and so on.

4. Bonus – Combinations of the above, i.e the most popular visited page with a conversion point, sorted by lowest conversion percentage with theoretical greatest chance for improvement.

Of course this is not the be all end all of what to look for or what to test in each area, but merely a good  refresher for those who need it, or a guiding hand for those confused with all the potential places to start testing first. But remember, it’s important to consider the opportunity costs in testing one area, page, path, etc. versus testing another.

Share

A marketing test hypothesis is a powerful and necessary part of your marketing optimization program when running tests. I am going to take you through creating a simple hypothesis.

A hypothesis clearly states:

  • What you are testing 
  • What your control and experimental groups are 
  • What outcome you predict will happen (based on an educated judgment) 
  • What is the alternate outcome 
  • What you will need to track specifically in order to prove or disprove your test prediction.

 
Let’s look at a hypothesis more closely.

A hypothesis is clear and specific, testable, and can be proved right or wrong.

Look at these differences between a prediction, a question, and a hypothesis:

A Test Prediction: Not asking for a phone number on my registration form will increase registrations.    
A prediction is the outcome you expect – more or less your educated guess of what will happen.

A Test Question: Will not asking for a phone number on my registration form increase registrations?

A Test Hypothesis: Paid Search traffic reaching my registration form that does not ask for a phone number will  produce more registrations than Paid Search traffic reaching my registration form that asks for a phone number.

A hypothesis states with conviction what results you expect to see from your test, both from your control and your experimental group – it is here that you will state your test prediction.  And since we know that a hypothesis has to be able to be proven either right or wrong, we only have 2 possible outcomes – either my registration form that does not ask for a phone number increases registrations over my control that does ask for it, or it doesn’t produce more registrations(or a tie).


And finally, let’s break down the above marketing test hypothesis to show specifically what it explains:

  • Paid Search traffic reaching my registration form that does not ask for a phone number is the experimental group
  • Produce more registrations is the outcome we expect from the experiment and what we want to track
  • Paid Search traffic reaching my registration form that asks for a phone number is the comparison or control group.

If you wanted to run this test with multiple panels each having a different form field requirements (size), you could in reality replace phone number with less fields:

Paid Search traffic reaching my registration form that has less than 7 fields will produce more registrations than Paid Search traffic reaching my registration form that has 7 fields. ” 

You are still testing your hypothesis of that less fields will produce more registrations than your control of 7 fields, but you will determine from your testing which length is optimal to rollout with if your hypothesis is true.

With your completed hypothesis you can now execute your marketing test to your website visitors and let them prove or disprove it. If they prove your hypothesis to be true then you did a great job with your hypothesis’s prediction, if they disprove it, you still have learned something (make sure you take away lessons from each test!) – The test is not a failure, just try again after formulating a new hypothesis.

Share

It’s imperative to document every effort of your online testing and optimization program. Not only to see the progression of improvement over time, but to also have reference for future tests you are planning or questions that may arise from others you are working with.

How I document my A/B and multivariate testing is as follows:

First, I primarily use Excel for all my documentation efforts due its ease of use.

I create a Master Test spreadsheet that serves as a very top-level summary for quick glancing. This spreadsheet is constructed to have column headers for the following:

  • Test name – giving each test a descriptive and unique name
  • Test date range – documenting the start and finish date of my tests
  • Test hypothesis – why I am running this test and why I believe my outcome will be such
  • Results – brief description of the result

Each row is for a single test. I then link the Test Name cell for each test to another spreadsheet that is built specifically for that test (If you have ten tests in the Master Test spreadsheet each test would link to its own Individual Test spreadsheet for a total of 10 spreadsheets plus the Master Test Sheet).

Each Individual Test sheet contains multiple tabs and information, and this is where the detail will go throughout the test.

The first tab contains the test information and summary and broken into sections:

  • Test name
  • Test date range
  • Test hypothesis
  • What type of test (a/b, multivariate), and how many panels or combinations
  • Traffic data (source of traffic, and current traffic stats)
  • What type of metrics I will be using to determine the results and how to determine the metrics (is conversion impressions divided by sales, or impressions divided by clicks on a certain button etc?)
  • A space to record final metric results (control performed as such, top performers identified individual performed as such)
  • Learning’s (both as the test is live and from the results)
  • Ideas for future tests based off of this information
  • Next actions (will this be rolled out etc.)
  • Miscellaneous notes

 

I also have other tabs in the Individual Test spreadsheet:

  • Screenshot of control
  • Screenshots of test panels or combinations (depending on how many there are). If there are too many panels or page combinations to take screenshots of, after the test is ended I take screenshots of the top performing test panels for future comparisons)
  • Screenshots of Test statistics (When I am using Google Website Optimizer I take daily screenshots of the stats admins. and store in a separate folder, but the final screenshot from the point at which we end the test is stored in the Individual Test spreadsheet – just in case I transcribe something wrong I have an actual reference to go back to.
  • Various other tabs as necessary for reference such as more detailed metrics information, etc.

 

I also keep a folder for each test (with the folder using the test name) that contains my test spec PowerPoint so that I can see all of the elements or options that we are using, analytics data, screenshots of everything-basically anything used from the conception of the test  all the way through to the end.

What this enables me to do is at any point in time have a huge history of each test both from a visual standpoint and data-driven standpoint. The Master Test sheet gives me quick access to the individual tests but also a timeline of the testing I have done.

Testing Documentation Flow

My Testing Documentation Flow

Share