Use optimizely for A/B Testing

Abstract:

Optimizary is a website that provides a/B testing services. It allows you to visually edit test content and objectives online, which is simple and convenient.

1. Introduction to A/B Testing

To put it simply, a/B testing is to develop two solutions (for example, two pages) for the same target, so that some users can use solution A and others can use solution B, record the user's usage to see which solution is more in line with the design goal. Refer:

Http://oldj.net/article/ab-testing-basic-concept

Http://www.zhihu.com/question/20045543

Http://en.wikipedia.org/wiki/A/ B _testing

Https://www.optimizely.com/ab-testing

2. What is optimizary?

Optimizary is an Internet company that provides a/B testing SaaS services. The site is https://www.optimizely.com. Founded in 1.2, the company earned $2013.04 M, $28 m for a round, and $57 m for B round. Refer:

Https://www.optimizely.com/about

Http://en.wikipedia.org/wiki/Optimizely

Http://www.zhihu.com/question/23756936

3. Brief Introduction to optimizary

(1) test and release

Test release process: log on to the optimizary website and enter the URL to be tested. On the optimizary page, edit the compare version. On the optimizary page, set the test target. On the page source code, add a small JS section, this JS is generated by optimizary-> start the test.

All test and release operations are visualized. You can edit the content, coordinates, and styles of each element on the page online, or insert or delete an element. When adding images, you can upload them to the optimizary CDN, or provide URLs for existing images.

Multiple test targets are set:

A) Click Event, and visually configure the click page on a preview page.

B) pageview, which provides four URL matching methods: precise, simple, substring, and regular expression

C) custom events should be set through the API provided by optimizary.

Operation details see: https://www.optimizely.com/gettingstarted

The key to test and release is the JS Code generated by optimizary. All online editing related to the test will be reflected in the JS Code, and the JS control page display and data statistics will finally be used.

 

Note: This JS must be added to the HTML head. This JS is saved on the CDN server of optimizary. Each test has a fixed js url, but a new JS will be generated every time you modify the test content or target, but the URL remains unchanged.

The actual huge page revision is not suitable for online page editing. There will be actual page files on our own server. You can perform a/B test on the old and new page versions through redirection, see: https://help.optimizely.com/hc/en-us/articles/200040675-Redirect-experiments-Comparing-two-separate-pages

(2) page browsing and statistics

Page browsing process: Enter the URL and press ENTER-> download page elements, and download the Js of optimizary-> Js of optimizary to calculate the page version displayed to the user through hash, and send a UV record to the optimizary server through JS control display-> user operation... -> If a preset event is triggered, the corresponding records are sent to the optimizary server through Ajax.

When you browse the test page, two additional types of traffic are generated:

A) download the JS Code of optimizary: Because JS is in the head, you must first load this JS when opening the page, which will affect the page opening speed. The JS file size is 46kb and is located on a server in the United States. The speed of opening my network is between-MS. You can consider putting this JS on your own domestic server, but ensure that the JS file should be updated in a timely manner during each modification test.

B) trigger a statistical event and send a message to the optimizary server: the server that receives the operation log is also in the United States. My network latency is over ms, and the sending failure or long time may occur at a low probability, however, it does not affect users because operation logs are sent asynchronously. If sending fails, the statistics are lost.

(3) view statistical results

Optimizary has a dedicated test result page, which is easy to use and intuitive. There are several indicators, which are described as follows:

• Visitor over time: UV count of the statistical cycle

• Conversion over time: If you perform operations on the page, the value is 1. If you perform multiple operations, the value is 1.

• Conversion rate: calculation result of conversion/visitor

• Improvement: select a version called baseline, which indicates the conversion rate difference from baseline.

• Chance to beat baseline: If baseline is replaced with the current version, the probability of victory is exceeded.

4. Some technical details of optimizary

(1) how to identify UV

The key lies in cookie, which stores information such as user ID, Test ID, browser version record, User device information, last 10 historical operations, and timestamp. For details, see:

Https://help.optimizely.com/hc/fr/articles/200040335-What-information-does-Optimizely-store-in-its-cookies-

Https://help.optimizely.com/hc/en-us/articles/200040335-How-Optimizely-Works-Snippet-order-of-operations-JavaScript-evaluation-timing-and-cookies

In the JS generated by optimizary, based on user cookies, a hash algorithm is used to calculate the page style displayed by the current user or jump to the page. A user will always see pages of the same version, the details of the algorithm cannot be found. If the cookie is deleted, even if the same browser on the same computer is accessed again, it will be treated as a new UV.

(2) optimizary Architecture

There are three components:

A) Google App Engine, used to store user accounts and test configuration information

B) Amazon Elastic Compute Cloud and simple storage service for receiving, storing, and computing statistics

C) Akamai, used to store generated JS files

No more technical details are disclosed on the official website of optimizary.

5. optimizary's pitfalls

(1) Fee

The A/B test is charged according to the UV of the website to be tested. Similar to the carrier package, the test is charged on a monthly basis. Different packages provide different monthly UV ceilings. Additional charges are calculated if the quota is exceeded. See: https://www.optimizely.com/pricing

(2) statistical accuracy

Possible causes of Inaccurate statistics are:

A) failed to send the statistical event: the statistical server is located in the United States, and the sending fails at a low probability, as described earlier in this document.

B) non-real-time statistical results: After a user operation, the statistical results can be found in the statistical results after 5 minutes, and the indicators are calculated separately, for example, the user's UV number may already exceed 1, but the user's click operation does not include more than 1

C) optimizary may not be correct.

(3) It is best for the optimizary website to flip the XX Wall

Pages such as test configuration and statistical results are often unavailable. It is best to bring a ladder.