Sample Scenario:

• You want to add a page with a video or demo to your web site.
• You believe that it is important for visitors to view as much as possible of the video.
• You have 2 videos to test against each other, and your test goal is to determine which one of your videos keeps more visitors watching (engaged) for at least a certain period of time and plan to  keep the one that engages viewers more than the other.

In this scenario, since your test goal is to determine which video users are more engaged in watching for at least a certain period of time you will therefore want a conversion to trigger and to be recorded in GWO after that set period of predetermined minimum viewing time.

Back to the Scenario:

• You want a conversion to trigger if a visitor in the experiment views at least 20 seconds of your 1 minute videos.

This is easily done by using Time On Page as your conversion goal. What basically happens is that if the visitor stays on the page or in this scenario views the video for that certain period of time that you have predetermined; the setTimeout JavaScript function (in the script given further down in the article) executes the script that reports a conversion in Google Website Optimizer.

To do this, use this time on page goal tracking script and place it at the bottom of the page that has your control script and test page tracking script. Change the red bold faced font in the script below to the amount of time you want the visitor to have reached before a conversion is triggered. The number is represented in milliseconds where 1000 milliseconds equal 1 second – so in this scenario since we want 20 seconds of viewing time to trigger a conversion, simple multiplication of 1000 milliseconds x 20 = 20000. Therefore if the viewer views the video for AT LEAST 20 seconds a conversion is recorded. You will replace the xxxx’s in blue below with the data given to you in the control script and test page tracking script (it references your GWO account and this test).

<script>
if(typeof(_gat)!=’object’)document.write(’<sc’+'ript src=”http’+ (document.location.protocol==’https:’?’s://ssl’:'://www’)+ ‘.google-analytics.com/ga.js”></sc’+'ript>’)
</script>

<script>
setTimeout(’_gat._getTracker(”UA-xxxxx-x“)._trackPageView(”/xxxxxxxxxx/goal”)’, 20000);
</script>

Note: Since the conversion is triggered in this instance by a viewer who stays on your page for a certain minimum period of time, and if your page has more than just the video (such as news, photos, or articles) and a visitor stays for the minimum period of time doing other “things”, the conversion will still trigger. You either need to set your test to be just for the video (requiring a more advanced setup), or have nothing else on the page but the video – and by nothing else I am referring to other pieces of content that could potentially keep people on that page doing something other than viewing the video.

Related posts:

1. How to Interpret the Estimated Conversion Rate Range in Google Website Optimizer Interpreting the Estimated Conversion Rate Range Properly in GWO is...

• Ability to add notes to tests – it would useful to be able to make notes about tests within the test admin itself for each test. Many times we have multiple people looking at a test and I would like to leave comments and get responses within the test, not only for ease but for permanent documentation

Today, Trevor Claiborne of the Google Website Optimizer team posted on the GWO Blog that they have added the ability to add notes to your GWO experiments. I am so very excited as this should really help out with collaboration and to help others who like to look at the current status of tests we have live as even the documentation for past tests. Although I already have a reliable system for test documentation this should help decrease the amount of cross-referencing and multiple email responses to others (I hope!) that now can easily be documented within the test itself.

Thank you Google Website Optimizer team!

No related posts.

It’s easy to get initially excited when you see that one of your test panels in Google Website Optimizer has a higher Estimated Conversion Rate than that of your control panel as presented on the Combination Report page. This may even lead to you believe you can end the test (prematurely I may add).

Unfortunately, just looking at the Conversion Rate number given to you by Google in the bold type font isn’t enough, you most certainly need to do a little bit of visualization to really have a better understanding of what is going on and how they are performing against each other.

Whether you are running an A/B test or a multivariate test, this is important to know for either – the number they give you is a conversion RATE RANGE. Many people mistakenly look at just the number given and do not visualize the full conversion range given along with it (done so with simple addition and subtraction of the number given next to the estimated conversion rate after  the plus and minus sign). This range is based on the observed conversion rate of during the experiment thus far. Not factoring this in can lead to many people ending or wanting to end tests before they are truly ready to be ended. For example,

Estimated Conversion Rate

• Test Panel – 6.0% +/- 1.0%
• Control Panel – 5.5% +/- 1.0%

Reading and interpreting this correctly would actually tell you that the:

• Test Panel is converting in the range of 5.0% to 7.0%, and the
• Control Panel is converting in the range of 4.5% to 6.5%

This being true, their conversion rate ranges are overlapping each other.  Visualizing this information shows you the overlap much more clearly as shown below:

So, is the test panel really converting better than the control? Potentially the control panel could still be converting better – it’s not clear enough to make a decisive or correct decision at this point.

Now think of a scenario such as two panels having a higher conversion rate and much larger conversion rate range:

• Test – 55.0% +/- 8.0% (conversion rate range of 47.0% to 63.0%)
• Control – 52.0% +/- 7.0% (conversion rate range of 45.0% to 59.0%)

Some may feel that the test panel is the winner and so it’s ok to stop the test and roll out with the test panel…BUT…

As the test goes on for a longer period of time, you should find the conversion rate range getting “tighter” and the above test potentially ending in such as a way as:

• Control panel converting at 58% +/- 1 .0%, and the
• Test panel converting at 47.0% +/- 1.0%.

In this ending scenario the control panel is actually the winner not the test panel as we saw mistakenly earlier on. Ending the test too early certainly changes the story drastically as does the results could differ as well!

I know sometimes it’s exciting to see your test panel winning, but patience is certainly of high importance in A/B and multivariate testing in order to make sure your tests and results are always moving forward correctly.

Related posts:

1. Testing Video Engagement Using Google Website Optimizer Using Minimum Viewing Time as a A/B Test Conversion Goal...

Please make sure that when you fill out the calculator with your test information that you then click on the calculate results button and look below it in the results section to see if you have a winner or not.

If the A/B calculator says that you have a winner, you can determine the winner very easily by just looking at the conversion rates of  each of the panels and the one with the better conversion rate is your winner. If the A/B split test calculator results declare that you don’t have a winner then you will need to run your test longer, or run a different test.

You can try out my online A/B split test calculator at www.SplitTestCalculator.com .

No related posts.

The Combinations report provides you with the performance data for all of your page combinations that you are testing and compares each page combination to your original page (the control) in order for you to determine what test page combinations are outperforming the original.

The Page Sections report on the other hand displays all of your page sections that you are testing and the elements within each section in order to show you the best performing page element for each particular section.

Here is the scoop on the Page Sections report.

Many times, when you are running a test you will notice that the best performing elements according to the Page Sections report are not the same elements that are present in your winning page combination. It’s not as simple as looking at the different sections and choosing all the “best” elements according to Google Website Optimizer Page Sections report and then rolling out a page that contains these elements. The reason being is that the performance of the elements in each section being tested as shown in the Page Sections report are not being calculated and presented in context with the other sections. By this I mean that the results of each element in each section are presented by how they performed against the other elements in that same page section. It’s not taking into account the context of all the other elements on the page when showing you the best performers. To see which the best elements are in proper context, view your Combinations report and look at the individual elements in your top performing page combination.

Confused?

Let me give you an example. If you are testing a main headline, and have 3 different versions:

• Control/original headline
• Test headline # 1
• Test headline #2

And the report shows that test headline # 1 is the best performing element in that area, it’s only reporting to you that between those 3 headlines, headline #1 is the best performer, but it is not taking into consideration how it interacts with the other elements on the page. It’s not saying that for the page you are testing, headline #1 is the best to use for the roll-out page, but instead – between those 3 headlines, headline #1 is the best performing if you are comparing just the performance of the headline itself. Headline #1 might perform the best out of the 3 headlines when compared to each other out of context with the rest of the page elements, but Headline #2 might perform the best in combination with all the other elements and therefore is the one you want to roll out with (but not necessarily reported this way in the Page Sections report). So please, don’t use the Page Sections report to pick out which elements to roll-out live with.

The Relevance Rating graphic (currently found to the left of each page section), alerts you to how much impact each section had on your test (as defined by your test’s conversion goal). It’s presented as a range, from 0 to 5, with 0 representing that the section had virtually no impact on conversions and a 5 as having a high impact on page conversions.

So while the Page Sections report is important to look at to understand what is happening between the different elements in the page sections themselves, and the Relevance Rating is important to look at to understand what page sections are providing what level of impact to page conversions, you should ideally only use this information to help you gain more insights into the test. Insights that will also give you further help in planning future testing of the page.

However, you will only want to roll-out the panel that is deemed the best performer according to the page Combinations report. Of course this is after you finish running your head-to-head follow-up test between your control and the best performing page combination to verify your results outside of the multivariate testing environment!

Related posts:

1. How to Interpret the Estimated Conversion Rate Range in Google Website Optimizer Interpreting the Estimated Conversion Rate Range Properly in GWO is...
2. Testing Video Engagement Using Google Website Optimizer Using Minimum Viewing Time as a A/B Test Conversion Goal...