[Machine learning] How to choose model--cross validation

For a machine learning system, there are several problems to be solved:

1, how to choose Feature.

2, which algorithm to choose.

3, how to set the parameters for this algorithm.

Together, these questions are "how to choose a model".

For example: can realize the classification system algorithm has one-vs-all logistic regression,neural NETWORK,SVM and so on, we should use which one.

To solve this problem, we need to use different combinations (including algorithms, parameters, feature) on the data d_train to training, get different model.

To test these model, you need to use the new data D_CV. Then, the model of the classification of the forecast results to do a evaluation (how to evaluate. ）

Cross Validation

In the above two datasets are used, the training dataset D_train and test DataSet D_CV, this method is cross validation, take Movielens 100k DataSet as an example, the following steps:

1, get training data set

2, this data is divided into training set and cross validation set (CV set)

3, the use of different combinations (including algorithms, parameters, feature), the training set for training and get different model;

4, the model on the CV set on the performance of a score, choose a better performance models;

There is a need to note that we will eventually choose to perform the best model on the CV set, but the final evaluation of this model is to be in a new data d_test (similar to the Netflix Prize competition, The official eventually gives your model a rating of data) on the test. Andrew NG recommends dividing the data as follows:

k-fold Cross validtion

The procedure is as follows:

1, select 20% of the data as Test set, put aside;

2, divide the remaining data into K equal parts (k-fold);

3, select 1 fold of data as D_CV, the remaining k-1 fold data as d_train, so you can get K (D_train, D_CV) data

4, for an algorithm, so that it in K D_train training to get K model, and then use d_train corresponding D_CV to do the test evaluation of the model, to K evaluation average

5, for each algorithm, repeat the 4th step, finally choose a evaluation of the best model

Figure 1,k-fold Schematic

In practice, 5-fold,10-fold is a better choice.

Resources:

1, the cornerstone of machine learning-the 15th lecture, Validation---coursera

2,machine Learning-10,model Selection and Train/validation/test Set---coursera

3,practical Machine Learning-week1,cross Validation