Model Evaluation and Model Selection

Machine learning algorithm principle, implementation and practice-model evaluation and Model Selection

 

1. Training error and test error

The purpose of machine learning is to make the learned model have good prediction ability for both known data and unknown data.

Assume that the learned model is $ y =\hat {f} (x) $, and the training error is model $ y =\hat {f} (X) $ average loss of the Training Dataset:

$ R _ {EMP} (\ hat {f}) =\ frac {1} {n} \ sum _ {I = 1} ^ NL (y_ I, \ hat {f} (x_ I) $

$ N $ indicates the capacity of the training sample.

The test error is the average loss of the model $ y =\hat {f} (x) $ about the test Dataset:

$ E _ {test} (\ hat {f}) =\ frac {1} {n'} \ sum _ {I = 1} ^ NL (y_ I, \ hat {f} (x_ I) $

$ N' $ indicates the test sample size.

When the loss function is 0-1, the test error becomes the error rate on the common test dataset (the number of prediction errors divided by the total number of test data ).

The size of the training error makes sense for determining whether a given problem is easy to learn, but it is not important in essence. The test error reflects the prediction capability of the Learning Method on unknown datasets and is an important concept in learning. Obviously, given two learning methods, the method with low test error has better prediction ability and is a more effective method. The ability of learning methods to predict unknown data is usually calledGeneralization ability(Generalization ability ).

2. Over-fitting and Model Selection

We know that there are infinite models in Space Theory, and they have different complexities (generally expressed as the number of parameters). We hope to select or learn a suitable model.

If you increase the prediction capability of training data, the complexity of the selected model is usually higher than that of the real model. This phenomenon is calledOverfitting. Overfitting means that the model selected during learning contains too many parameters, so that this model has a good prediction for known data but has a poor prediction for unknown data.

The following uses polynomial function fitting as an example to describe over-fitting and model selection. This is a regression problem.

Now a training dataset is given:

$ T =\{ (x_1, y_1), (X_2, y_2), \ dots, (X_n, Y_n) \}$ $

$ X_ I \ In r$ is the observed value of Input $ x $, and $ y_ I \ In r$ is the observed value of output $ y $. The task of polynomial function fitting is to assume that the given data is generated by the M-degree polynomial function, and select the M-degree polynomial function that is most likely to generate the data, that is, select a function in the M-degree polynomial function that has many predictive capabilities for both known and unknown data.

We use $ Y = sin (x) $ to generate 10 data points, and add some errors to the value of $ y $ as appropriate. Next we use 0 ~ The data is fitted with 9 polynomials.

The polynomial fitting conditions for $ M = 1, m = 3, M = 9 $ are given. When $ M = 1 $, the polynomial curve is a straight line, and the data fitting effect is poor. Conversely, if $ M = 9 $, the training error is 0 if the polynomial curve passes through each data point. From the perspective of fitting the given training data, the effect is the best. However, due to the noise in the training data itself, this fitting curve often does not have the best ability to predict unknown data. In this case, over-fitting will happen.

import numpy as npimport matplotlib.pyplot as pltimport randomx = np.linspace(0,1,10)y = np.sin(2*np.pi*x)for i in range(0,10):        y[i] = y[i] + random.uniform(-0.4,0.4)p = np.polyfit(x,y,9)t = np.linspace(0,1.0,100)plt.plot(x,y,‘o‘)plt.plot(t,np.sin(np.pi*2*t),label=‘$y=sin(x)$‘);plt.plot(t,np.polyval(p,t),label=‘$y = \sum_{i=0}^Mw_ix_i,M=9,x_0=0$‘);plt.legend()plt.show()

3. Regularization and cross-validation 3.1 Regularization

When introducing machine learning strategies, the previous article mentioned the concept of minimizing structural risks. Structural risk minimization is a strategy proposed to solve the problem of over-fitting. It adds a regularization item to the empirical risk. Normalization items are generally monotonically incrementing functions of model complexity. The more complicated the model is, the larger the regularization value is. For example, the regularization item can be the norm of the vector of the model parameter.

Normalization items can be in different forms. For example, in Regression Problems, the loss function is a square loss, and the regularization item can be the $ L_2 $ norm of the parameter vector:

$ L (w) =\frac {1} {n} \ sum _ {I = 1} ^ N (f (x_ I; W)-y_ I) ^ 2 + \ frac {\ Lambda} {2} | w | ^ 2 $

Here, $ | w | $ represents the $ L_2 $ norm of the parameter vector $ W $.

The regularization item can also be the $ L_1 $ norm of the parameter vector:

$ L (w) =\frac {1} {n} \ sum _ {I = 1} ^ N (f (x_ I; W)-y_ I) ^ 2 + \ Lambda | w | _ 1 $

Here, $ | w | _ 1 $ indicates the $ L_1 $ norm of the parameter vector $ W $.

Regularization complies with Occam's Razor principle. The idea of the Occam Razor application in model selection is: In all the models that may be selected, it is the best model to explain the known data well and be very simple, that is, the model to be selected. From the perspective of Bayesian estimation, regularization items correspond to the prior probability of the model. It can be assumed that a complex model has a lower anterior probability, while a simple model has a higher anterior probability.

3.2 cross Verification

If the given sample data is sufficient, a simple method for model selection is to randomly split the dataset into three parts: training set, validation set, and test set. The training set is used to train the model, the validation set is used to select the model, and the test set is used to ultimately evaluate the learning method. In the model with different complexity learned, select the model with the minimum prediction error for the verification set.

However, in many practical applications, data is insufficient. To select a model, you can use the cross-validation method. The basic idea of cross-validation is to use data repeatedly. Split the given data and combine the split dataset into a training set and a test set, on this basis, training, testing, and model selection are performed repeatedly.

1. Simple cross-validation

First, the data is randomly divided into two parts: one part as the training set and the other part as the test set. Then, the training set is used to train the model under various conditions to obtain different models; evaluate the test error of each model in the test set, and select the model with the smallest test error.

2. S-fold crossover Verification

This method is most widely used. First, the given data is randomly divided into S subsets of the same size, and then use the data Training Model of the subset of the S-1, and then use the remaining subset to test the model; this process repeats the possible S selection, and finally selects the model with the smallest mean test error in the s evaluation.

3. Leave a cross verification record

The feature of S-fold cross-validation is $ S = N $, which is called leave a cross-validation, which is often used when data is lacking. Here, n is the capacity of the given dataset.

4. generalization ability

The generalization ability of the learning method refers to the ability of the model learned by this method to predict unknown data. It is an essential feature of the learning method. In reality, the most widely used method is to evaluate the generalization ability of the learning method by testing the error of the dataset. However, because the data is limited and does not represent all unknown samples, it is very likely that the evaluation result is unreliable.

Next we will analyze the generalization ability of the learning method theoretically.

First, the definition of generalization error is given. If the learned model is $ \ hat {f} $, the prediction error of unknown data using this model is generalized error)

$ R _ {exp} (\ hat {f}) = E_P [L (Y, \ hat {f} (x)] = \ int _ {\ mathcal {x} \ times \ mathcal {y} l (Y, \ hat {f} (x) p (x, y) dxdy $

Generalization error reflects the generalization ability of the learning method. If the model learned by one method has a smaller generalization error than the model learned by another method, this method is more effective. In fact, generalized error is the expected risk of the learned model.

Model Evaluation and Model Selection