A/B testing: Basic Concepts

In website design, we often face the choice of multiple design schemes, such as whether a button is in red or blue, left or right. The traditional solution is usually a collective discussion or vote, or an expert or leader can decide whether or not to choose a random online solution. Although the traditional solution is effective in most cases, a/B testing (a/B-testing) may be a better solution to such problems.

To put it simply, a/B testing is to develop two solutions (for example, two pages) for the same target, so that some users can use solution A and others can use solution B, record the user's usage to see which solution is more in line with the design goal. Of course, there are still many details to be aware of in the actual operation process.

A/B testing is not a new method of Internet testing. In fact, there are also events similar to a/B testing in nature, such as dalavenque.

Dalavenque lives mainly on an island named Isabela in Galapagos in the Eastern Pacific Ocean. Some live in the western part of the island and the other in the eastern part of the island, due to the nuances of the living environment, they evolved into different beaks. This is considered an important example of natural selection theory.

Which of the following is more suitable for survival? Nature has given her solutions, allowing the birds to mutate themselves (multiple design schemes) and then survive. In the example of dalavenque, different environments have different solutions.

Although the above example has nothing to do with website design, it contains the core idea of a/B testing, that is:

1. parallel testing of multiple solutions;

2. Each solution has only one variable (for example, a bird's bill;

3. Rule-based survival of the fittest.

Note that the 2nd point implies the application scope of A/B testing, which must beSingle Variable. Sometimes multiple of our design drafts may be very different. In this case, it is generally not suitable for a/B testing, because there are too many variables and there will be a lot of interference between variables, it is difficult for us to use the/B testing method to find out the extent to which each variable affects the result. For example, potato roast meat and bean curd braised fish soup are both delicious, but it is difficult for us to compare which of the following has a greater impact on food, while potato roast meat and bean curd roast meat are a good comparison. In addition, although the name of A/B test only contains a and B, it does not mean that it can only be used to compare the two schemes. In fact, you can design multiple schemes for testing, the name "A/B testing" is just a habit.

Back to website design, in general, each design scheme should be basically the same, but it is different in a certain place, such as a layout, text, image, color, etc. Then, different solutions are displayed for different users.

It should be noted that different users should always see the same solution during one of their browsing processes. For example, if he saw solution a at the beginning, he should always show solution a to him in this session, instead of letting him see solution a later, and then let him see solution B later. At the same time, you also need to control the number of users accessing each version. In most cases, we want to evenly distribute visitors to different versions. To do this, it is easy to determine which version to Display Based on the cookie (such as the last digit of the Cookie Session ID.

The following is a/B testing:

We can see that to implement a/B testing, we need to do the following:

1. develop and deploy two (or more) different versions;

2. Collect data;

3. analyze the data and obtain the result.