The meaning of training sets, validation sets, and test sets

Original

In supervised machine learning, it is often said that training sets (train), validation sets (validation), and test sets (tests), the distinction between the three sets can be confusing, and in particular, some readers do not know the difference between a validation set and a test set. I. Division

If we ourselves already have a large annotation data set, want to complete a supervised model of the test, then usually using a uniform random sampling method, the data set is divided into training sets, validation sets, test sets, the three sets can not have intersection, the common proportion is 8:1:1, of course, the proportion is artificial. From this point of view, three sets are all in the same distribution.

If it is a game, the official only provides a labeled DataSet (as a training set) and a test set that is not labeled, so when we do the model, we usually manually divide a validation set from the training set. At this time we usually no longer divide a test set, there are two possible reasons: 1, the game is basically very key, the training set of the sample is less; 2. We cannot guarantee that the test set to be submitted is fully distributed with the training set, so it is not significant to partition a test set that is distributed with the training set. II. Parameters

With the model, the training set is used to train parameters, said the exact point, is generally used to reduce the gradient. The validation set is basically used to test the accuracy of the current model once each epoch is completed. Because the validation set has no intersection with the training set, this accuracy is reliable. So why do you need a test set?

This requires a distinction between the various parameters of the model. In fact, for a model, its parameters can be divided into ordinary parameters and super-parameters . Without the introduction of reinforcement learning, the normal parameters can be updated by gradient descent, which is the parameter updated by the training set. In addition, the concept of hyper-parameters, such as network layer, network nodes, iterations, learning rate, and so on, these parameters are not in the gradient drop in the update range. Although there are already some algorithms that can be used to search for the parameters of the model, in most cases we will manually adjust them according to the validation set. III. So

That is to say, in the narrow sense, the validation set does not participate in the process of gradient descent, that is, is not trained; but in a broad sense, the validation set is involved in a "manual parameter" process, we adjust the results of the validation set by the iterative algebra, adjust the learning rate and so on, so that the results in the validation set optimal Therefore, we can also assume that the validation set is also involved in the training.

So obviously, we also need a completely untrained set, that is, the test set, we do not have to test the set gradient drop, nor it to control the hyper-parameters, but after the final training of the model to test the final accuracy rate. Iv. however

The smart reader will be analogous to the fact that this is an endless process. If the test set accuracy is very poor, then we will still adjust the parameters of the model, it can be considered that the test set is also involved in training. Well, we might also need a "test test set", and maybe "test test set" ...

Forget it, or stop it in the test set.