[Comprehensive] training set validation set Test Set)

In supervised machine learning, datasets are often divided into 2 ~ Three, namely, the training set validation set test set ).

Http://blog.sina.com.cn/s/blog_4d2f6cf201000cjx.html

Generally, samples must be divided into three independent training sets, validation sets and test sets ). The training set is used to estimate the model, and the validation set is used to determine the network structure or control the parameters of the model complexity. The test set tests the final performance of the optimal model. A typical division is that the training set accounts for 50% of the total samples, while the others account for 25%. The three parts are randomly extracted from the samples.
When there are few samples, the above division is not suitable. It is often used to keep a small part of the test set. Then the K-fold cross-validation method is used for the remaining n samples. Is to disrupt the sample, and then evenly divided into k parts, take turns to select the K-1 training, the remaining one to verify, calculate the sum of squares of prediction errors, finally, the sum of squared values of K prediction errors is used as the basis for selecting the optimal model structure. The Special K is to take N, that is, leave a method (leave one out ).

Http://www.cppblog.com/guijie/archive/2008/07/29/57407.html

These three terms are extremely common in machine learning, but many people are not very clear about their concepts, especially the latter two are often mixed. Ripley, B .d (1996) defines the three words in his classic monograph Pattern Recognition and neural networks.
Training set: a set of examples used for learning, which is to fit the parameters [I. e., weights] of the classifier.
Validation Set: a set of examples used to tune the parameters [I. E ., architecture, not weights] of a classifier, for example to choose the number of hidden units in a neural network.
Test Set: a set of examples used only to assess the performance [generalization] of a fully specified classifier.
Apparently, the training set is used to train the model or determine the model parameters, such as the weight value in Ann. The validation set is used for model selection ), that is, the final optimization and determination of the model, such as the structure of Ann; while the test set is purely to test the promotion capability of the trained model. Of course, test set does not guarantee the correctness of the model. He just said that similar data will use this model to produce similar results. In practice, however, datasets are generally divided into two types: training set and test set. Most articles do not involve validation set.
RIPLEY also talked about why separate test and validation sets?
1. The error rate estimate of the final model on validation data will be biased (smaller than the true error rate) since the validation set is used to select the final model.
2. After assessing the final model with the test set, you must not tune the model any further.

Http://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set

Step 1) Training: each type of algorithm has its own Parameter options (the number of layers in a neural network, the number of trees in a random forest, etc ). for each of your algorithms, You must pick one option. that's why you have a validation set.

Step 2) validating: you now have a collection of algorithms. you must pick one algorithm. that's why you have a test set. most people pick the algorithm that performs best on the validation set (and that's OK ). but, if you do not measure your top-memory Ming algorithm's error rate on the test set, and just go with its error rate on the validation set, then you have blindly mistaken the "best possible scenario" for the "most likely scenario. "that's a recipe for disaster.

Step 3) testing: I suppose that if your algorithms did not have any parameters then you wocould not need a third step. in that case, your validation step wocould be your test step. perhaps MATLAB does not ask you for parameters or you have chosen not to use them and that is the source of your confusion.

My idea is that those option in neural network toolbox is for avoiding overfitting. in this situation the weights are specified for the training data only and don't show the global trend. by having a validation set, the iterations are adaptable to where decreases in the training data error cause decreases in validation data and increases in validation data error; along with decreases in training data error, this demonstrates the overfitting phenomenon.

Http://blog.sciencenet.cn/blog-397960-666113.html

Http://stackoverflow.com/questions/2976452/whats-is-the-difference-between-train-validation-and-test-set-in-neural-networ

For each epoch
For each training data instance
Propagate error through the network
Adjust the Weights
Calculate the accuracy over training data
For each validation data instance
Calculate the accuracy over the validation data
If the threshold validation accuracy is met
Exit Training
Else
Continue training

Once you're finished training, then you run against your testing set and verify that the accuracy is sufficient.

Training set: this data set is used to adjust the weights on the neural network.

Validation Set: this data set is used to minimize overfitting. you're not adjusting the weights of the network with this data set, you're just verifying that any increase in accuracy over the training data set actually yields an increase in accuracy over a data set that has not been shown to the network before, or at least the network hasn' t trained on it (I. e. validation data set ). if the accuracy over the training data set increases, but the accuracy over then validation data set stays the same or decreases, then you're overfitting your neural network and you should stop training.

Testing set: this data set is used only for testing the final solution in order to confirm the actual predictive power of the network.

Validating set is used in the process of training. testing set is not. The testing set allows

1) to see if the training set was enough and
2) whether the validation set did the job of preventing overfitting. if you use the testing set in the process of training then it will be just another validation set and it won't show what happens when new data is feeded in the network.

 

 

Training set: a set of examples used for learning, that is to fit the parameters [I. e., weights] of the classifier.

Validation Set: a set of examples used to tune the parameters [I. E ., architecture, not weights] of a classifier, for example to choose the number of hidden units in a neural network.

Test Set: a set of examples used only to assess the performance [generalization] of a fully specified classifier.

The error surface will be different for different sets of data from your data set (batch learning ). therefore if you find a very good local minima for your test set data, that may not be a very good point, and may be a very bad point in the surface generated by some other set of data for the same problem. therefore you need to compute such a model which not only finds a good weight configuration for the training set but also shocould be able to predict new data (which is not in the training set) with good error. in other words the network shocould be able to generalize the examples so that it learns the data and does not simply remembers or loads the training set by overfitting the training data.

The validation data set is a set of data for the function you want to learn, which you are not directly using to train the network. you are training the network with a set of data which you call the training data set. if you are using gradient based algorithm to train the network then the error surface and the gradient at some point will completely depend on the training data set thus the training data set is being directly used to adjust the weights. to make sure you don't overfit the network you need to input the validation dataset to the network and check if the error is within some range. because the validation set is not being using directly to adjust the weights of the netowork, therefore a good error for the validation and also the test set indicates that the network predicts well for the train set examples, also it is expected to perform well when new example are presented to the network which was not used in the training process.

Early stopping is a way to stop training. there are different variations available, the main outline is, both the train and the validation set errors are monitored, the train error decreases at each iteration (backprop and brothers) and at first the validation error decreases. the training is stopped at the moment the validation error starts to rise. the weight configuration at this point indicates a model, which predicts the training data well, as well as the data which is not seen by the network. but because the validation data actually affects the weight configuration indirectly to select the weight configuration. this is where the test set comes in. this set of data is never used in the training process. once a model is selected based on the validation set, the test set data is applied on the network model and the Error for this set is found. this error is a representative of the error which we can perform CT from absolutely new data for the same problem.