Regularized Logistic Regression

The problem to be solved is that a bunch of training datasets with two features are provided. from the distribution of the data, we can see that they are not very linearly segmented, therefore, it is necessary to use higher-order features for simulation. For example, this program uses the 6 power of the feature value to solve the problem.

Data

To begin, load the files 'ex5logx. dat 'and ex5logy. dat 'into your program. this dataset represents the training set of a logistic regression problem with two features. to avoid confusion later, we will refer to the two input features contained in 'ex5logx. dat as and. so in the 'ex5logx. dat 'file, the first column of numbers represents the feature, which you will plot on the horizontal axis, and the second feature represents, which you will plot on the vertical axis.

After loading the data, plot the points using different markers to distinguish between the two classifications. the commands in MATLAB/octave will be:

 

x = load(‘ex5Logx.dat‘); y = load(‘ex5Logy.dat‘);figure% Find the indices for the 2 classespos = find(y); neg = find(y == 0);plot(x(pos, 1), x(pos, 2), ‘+‘)hold onplot(x(neg, 1), x(neg, 2), ‘o‘)

After plow.your image, it shoshould look something like this:

Model

The hypothesis function is

 

Let's look at the parameter in the sigmoid function.

In this exercise, we will assign to be all monomials (meaning polynomial terms) of and up to the sixth power:

To clarify this notation: We have made a 28-feature vector where

The loss function of the system after the rule item is added is:

 

Newton's method

Recall that the Newton's method update Rule is

1.Is your feature vector, which is a 28x1 vector in this exercise.

2.Is a 28x1 vector.

3.And are 28x28 matrices.

4.And are scalars.

5.The matrix following in the Hessian formula is a 28x28 diagonal matrix with a zero in the upper left and ones on every other diagonal entry.

 

After convergence, use your values of Theta to find the demo-boundary in the classification problem. The demo-boundary is defined as the line where

 

Code

% Load data CLC, clear, close all; X = load ('ex5logx. dat '); y = load ('ex5logy. dat '); % plot the data distribution diagram plot (x (find (Y), 1), x (find (Y), 2), 'O', 'markerfacecolor ', 'B') Hold on; plot (x (find (y = 0), 1), x (find (y = 0), 2), 'r + ') legend ('y = 1', 'y = 0 ') % add polynomial features to X by % calling the Feature Mapping function % provided in separate M-filex = map_feature (x (:, 1), x (:, 2 )); % projection to high-dimensional feature space [M, N] = size (x); % initialize fitting parameterst Heta = zeros (n, 1); % define the sigmoid functiong = inline ('1. 0. /(1.0 + exp (-z) '); % setup for Newton's methodmax_itr = 15; J = zeros (max_itr, 1 ); % Lambda is the regularization parameterlambda = 1; % Lambda = 0, 1, 10. modify this location and run it three times to get three results. % Newton's methodfor I = 1: max_itr % calculate the hypothesis function z = x * Theta; H = g (z); % calculate J (for testing convergence) -- Loss Function J (I) = (1/m) * sum (-y. * log (h)-(1-y ). * log (1-H) +... (lambda/(2 * m) * norm (theta ([2: end]) ^ 2; % calculate gradient and Hessian. G = (lambda/M ). * Theta; G (1) = 0; % extra term for gradient L = (lambda/M ). * eye (n); L (1) = 0; % extra term for Hessian grad = (1/M ). * x' * (H-y) + G; H = (1/M ). * x' * diag (h) * diag (1-H) * x) + L; % here is the actual update Theta = theta-H \ grad; end % plot the results % We will evaluate Theta * x over a % grid of features and plot the contour % Where Theta * X equals zero % here is the grid rangeu = linspace (-1, 1.5, 200); V = linspace (-1, 1.5, 200); Z = zeros (length (u), length (V )); % evaluate z = Theta * x over the gridfor I = 1: length (u) for j = 1: length (v) Z (I, j) = map_feature (U (I), V (j) * Theta; % the plot is not the curve between the loss function and the number of iterations, the endendz = Z'; % important to transpose Z before calling contour % plot z = 0% notice you need to specify the range [0, 0] contour (u, v, Z, [0, 0], 'linewidth', 2) % indicates the interface when the value is 0 on Z, because when the value is 0, the probability is 0.5, legend ('y = 1', 'y = 0', 'demo-boundary ') Title (sprintf (' \ Lambda = % G', lambda ), 'fontsize', 14) Hold off % uncomment to plot J % figure % plot (0: MAX_ITR-1, J, 'O -- ', 'markerfacecolor', 'R ', 'markersize', 8) % xlabel ('iteration'); ylabel ('J ')

 

Result

Regularized Logistic Regression