Regularization -- Linear Regression

This section describes how to use regularization items. In some models of machine learning, if there are too many parameters and too few training samples, the trained model is prone to over-fitting. Therefore, in the loss function of the model, the parameters of the model must be "penalized". In this way, these parameters will not be too large. The smaller the parameters, the simpler the model, the simpler the model, the less prone to over-fitting.

Regularized Linear Regression

From looking at this plot, it seems that fitting a straight line might be too simple of an approximation. instead, we will try fitting a higher-order polynomial to the data to capture more of the variations in the points.

Let's try a second th-order polynomial. Our hypothesis will be

This means that we have a hypothesis of six features, because are now all features of our regression. notice that even though we are producing a polynomial fit, we still have a linear regression problem because the hypothesis is linear in each feature.

Since we are fitting a 5th-order polynomial to a data set of only 7 points, over-fitting is likely to occur. to guard against this, we will use regularization in our model.

Recall that in regularization problems, the goal is to minimize the following cost function with respect:

The regularization parameter is a control on your fitting parameters. as the magnitues of the fitting parameters increase, there will be an increasing penalty on the cost function. this penalty is dependent on the squares of the parameters as well as the magnqueue. also, notice that the summation after does not include

 

The larger the Lamda, the simpler the trained model-the greater the penalty for the next item

Normal Equations

Now we will find the best parameters of our model using the normal equations. Recall that the normal equations solution to regularized linear regression is

The matrix following is an diagonal matrix with a zero in the upper left and ones down the other diagonal entries. (Remember that is the number of features, not counting the intecept term ). the vector and the matrix have the same definition they had for unregularized regression:

Using this equation, find values for using the three regularization parameters below:

A.(This is the same case as non-regularized linear regression)

B.

C.

Code

CLC, clear % load data x = load ('ex5linx. dat '); y = load ('ex5liny. dat '); % display raw data plot (X, Y, 'O', 'markeredgecolor',' B ', 'markerfacecolor', 'R ') % convert the feature value to the training sample matrix X = [ones (length (x), 1) x. ^ 2 X. ^ 3 X. ^ 4 x. ^ 5]; [m n] = size (x); n = n-1; % calculate the sidta parameter and draw the fitting curve Rm = diag ([0; ones (n, 1)]); % The Matrix after Lamda = [0 1 10] '; colortype = {'G',' B ', 'R '}; SIDA = zeros (n + 1, 3); % initialization parameter sidaxrange = linspace (min (x (:, 2), max (x (:, 2 )))'; hold on; for I = SIDA (:, I) = inv (x' * x + Lamda (I ). * RM) * x' * Y; % calculation parameter SIDA norm_sida = norm (SIDA) % norm evaluate the level 2 norm yrange of SIDA = [ones (SIZE (xrange) xrange. ^ 2 xrange. ^ 3 ,... xrange. ^ 4 xrange. ^ 5] * SIDA (:, I); plot (xrange', yrange, char (colortype (I) Hold onendlegend ('traning data ', '\ Lambda = 0',' \ Lambda = 1', '\ Lambda = 10') % use of escape characters to hold off

 

 

Regularization -- Linear Regression