Google Website Optimizer Report interpretation

　　

Some time ago with a blog several test pages to try the use of Google Website optimizer to do A/b test, after this time to collect some test data, thank you for your help click. In fact, always wanted to introduce Google this site Contrast experiment optimization tools, completely free and easy to operate, and in the near future, independent of Google Website Optimizer will disappear, this piece of functionality will be integrated into Google Analytics inside, It is believed that many GA users have found the new experiments feature in the content module, which is derived from the Google Website Optimizer, and there should be some adjustments and changes to the function and usage.

For the use of Google Website Optimizer is not so complex, using the creation of the Experiment Wizard combined with the tips and help of Web pages, ordinary users to create their own experiments are completely no problem, but the site may be a lot of different details of the need to pay attention to Or you need to think of ways to make the experimental process and monitoring results more effective. So there's no introduction to GWO tools using itself, I am more interested in the results of the experimental output report, which involves the interval of the results of the prediction, the test scheme to win the probability of the increase, and so on, these indicators are calculated by statistical methods, compared to direct comparison between the two groups of observations, the comparison results more scientific, more convincing.

First look at the GWO output report, here is a/b test, if the use of multivariate testing (MVT) Report There are subtle differences, but the indicators and statistical logic should be the same:

　　

I used GWO to implement the simplest A/b test, an original version and a test version, and the output report was primarily a comparison of the conversion rates set during the experiment. The line chart above shows the conversion trend of the original version and the experimental version, as of the current conversion shown at the far right of the table below (conv./visitors), from the indicator name, GWO to measure the conversion rate is the number of converted users, should use cookies to uniquely identify the user ( Here is only a few new in the blog a few simple test pages, so the amount of data is small, and conversion rate than normal sites are significantly higher.

So, here we focus on the 3 indicators in the red box in the chart below to see how they are calculated.

Estimating conversion rates

EST can be seen from red Box 1 in the figure. Conv. rate,gwo The present conversion rate is estimated by positive and negative intervals, and the possible confidence intervals of the current version conversion rate are obtained (see the previous article-parameter estimation and confidence interval), where the current conversion rate is used to estimate the overall conversion rate p, You can then calculate the overall standard deviation σ= sqrt (P (1-p)/n), such as the overall standard deviation of the original version is about 0.0540, and the test version of the standard deviation of about 0.0647, according to the zα/2xσ calculated in the table above the positive and negative 7.6% and 9.1%, So we can guess the gwo used ZΑ/2 about 1.4, this number I tested a few times during the test, the basic very stable, according to the Z-value table, the confidence interval of the approximate confidence in 84%, also do not know why GWO to choose this confidence degree.

At the top of the red box 1, we can see that gwo the test version into three classes based on the effect of the test, with green as the winning test version, yellow for an indeterminate test version, and red for the defeated Test version. Google only gives a simple description, suggesting that we have the option to use the version shown as green because they are very likely (and only possible) to be superior to the original version, and a red version is recommended to stop the test. Specific to the indicators need to achieve what level will show green or red, I did not go to verify that the use of experience or interested students can go to observe the next try.

More than the original version of the chance

Prior to the T-Test and the card-side test This article introduces the method of using the card-side test to compare the probability differences between the two distribution data. But the card-side test can only test the difference of the significance, not directly to explain the probability of a group of samples is more than the likelihood of more than another sample, Therefore, in the report of GWO did not use the card-side test, and the use of a single tail z test. When the number of samples exceeds 30, we typically use a Z-test instead of a T-Test to compare the difference in the mean between the two sets of independent or paired samples, since the single tail test is used only to prove that the probability of a group of samples is significantly greater than another group of samples. The formula for calculating Z statistics is as follows:

　　

According to the table data, the conversion rate of the original scheme and the test plan are 78.9% and 78% respectively, according to the formula S2=p (1-P) respectively, the variance of the two groups is 0.1665 and 0.1716, and the two groups of samples N are 57 and 41 respectively, and then the z= 0.1068, check Z-value table can get z=0.10 probability of 46.02%,z=0.11 is 45.62%, the table shows chance to Beat orig.=45.9%, between the two, using a similar statistical method to get the value, The deviation is due to intermediate precision processing.

Observed improvements

The observed improvement of red box 3, this index needs to combine red frame 1 of the conversion confidence interval to see, combined with bar chart can be more clear results, look at the show I use Excel to calculate the conversion rate of the similar bar chart:

　　

Here I use two auxiliary dashed lines, if the first is the original version, then the test version of all and the original version of the conversion rate of the estimated range of differences will be shown in color, and the observed improvement is the conversion rate of the shading interval difference. For example, the red section of the second bar in the graph shows a corresponding value of-4.2%, while the third should be -1.6%+0.6%=-1.0%, the negative range on the left plus the positive interval on the right, and the fourth is 2.9%. This makes the calculation of all the metrics in the GWO report clear.

In fact, most of the time we use tools to complete the process of analysis or testing, nor do we have to understand all the indicators at the bottom of the calculation logic, we only know the meaning of these indicators and the role of the analysis of specific problems in the rational use of these indicators can be. And sometimes we can only count some basic data, so how to use these basic data to get some valuable and persuasive analysis results requires some appropriate statistical methods, which is why this article interprets GWO output statements.

» This article uses the BY-NC-SA agreement, reprint please indicate the source: the Website Data analysis» "Google Website Optimizer report Explanation"