Deep Learning: Four (logistic regression exercise)

　　 Preface:

This section exercises the relevant content of the logistic regression, referring to the information for the Web page: http://openclassroom.stanford.edu/MainFolder/DocumentPage.php?course= Deeplearning&doc=exercises/ex4/ex4.html. The training sample given here is characterized by a score of two subjects for 80 students, a sample value of whether the corresponding classmate is allowed to go to university, if it is allowed to use ' 1 ', otherwise it is not allowed to use ' 0 ', this is a typical two classification problem. In this question, the 80 samples were given 40 of the positive and negative samples. And this section uses the logistic regression to solve, the result of the solution is actually a probability value, of course, by comparing with 0.5 can become a two classification problem.

Experimental data:

Download Data ex4data.zip

Experimental basis:

In the logistic regression problem, the logistic function expression is as follows:

　　

The advantage of this is that the output can be compressed between 0~1. The loss function in the logistic regression problem differs from the loss function in linear regression, which is defined as:

If the Newton method is used to solve the parameters in the regression equation, the iterative formula of the parameters is:

One of the first-order and Hessian matrix expressions is as follows:

May have some friends, not very clear how the above two formulas are deduced, the first formula is relatively simple is j to Theta derivative, only a derivative, the specific derivation process can refer to Newton method, exponential distribution family, generalized linear model-Stanford ML Public Lesson Note 4 in summary 2, The second formula is the derivative of the first, which is equivalent to the second derivatives of J. As to why it is required that the first derivative and the second derivative are not clear, we can refer to summary 2 and above.

　　some MATLAB functions:

　　Find:

is a vector that is found, and the result is a subscript number for the value of the Find function when the parentheses value is true.

　　Inline:

Constructs an inline function, much like the mathematical derivation formula we wrote on the draft paper. Parameters are usually used in single quotation marks, which is the expression of the function, if there are more than one argument, then separated by single quotation marks one by one description. For example: g = inline (' sin (alpha*x) ', ' x ', ' alpha '), then the two-tuple function is g (x,alpha) = sin (alpha*x).

　　Experimental Results:

The distribution map of the training samples and the classification interface curves learned:

The curve between the loss function value and the number of iterations:

Results of the final output:

It can be seen that when a child's first course is 20 points and the second course is 80, the child is not allowed to go to college for 0.6680, so if it is classified as a two, it means that the child will not be allowed to go to college.  

　　Experiment Code (available on the original page):

% Exercise 4--Logistic regressionclear all; Close all;    CLCX = Load (' Ex4x.dat ');  y = Load (' ex4y.dat ');% load data [m, n] = size (x); % data is m row N column% Add intercept term to XX = [Ones (M, 1), x]; % Plot the training data% use different markers for positives a nd negativesfigurepos = find (y); Neg = Find (y = = 0),% positive and inverse plot (x (POS, 2), X (pos,3), ' + ')% positive example with "+", inverse example with "o" Callout Hold Onplot (x (neg, 2), X (Neg, 3), ' O ') hold ONXL Abel (' Exam 1 score ') Ylabel (' Exam 2 score ')% Initialize fitting Parameterstheta = Zeros (n+1, 1);% Define the sigmoid functi Ong = inline (' 1.0./(1.0 + exp (-Z) ');  % Newton ' s methodmax_itr = 7;  J = Zeros (max_itr, 1); For i = 1:max_itr   % Calculate the hypothesis function    z = x * theta;    h = g (z);       % Calculate gradient and hessian.   % the formulas below are Equivalent to the summation formulas   % given in the lecture videos.    Grad = (1/m). *x ' * (h-y); &nb Sp   H = (1/m). *x ' *Diag (h) * DIAG (1-h) * x;       % Calculate J (for testing convergence)     J (i) = (1/m) * SUM (-y.*log (h)-(1-y). *log (1-h));        theta = theta-h\grad;end% Display thetatheta% calcula Te the probability that a student with% score in exam 1 and score on exam 2 % would not be admittedprob = 1-g ( [1, 80]*theta]% Plot Newton ' s method result% only need 2 points to define a line, so choose, endpointsplot_x = [min (X (:, 2))-2,  max (x (:, 2)) +2];% Calculate the decision boundary lineplot_y = ( -1./theta (3)). * (Theta (2). *plot_x + Theta (1));p lot (plot_x, plot_y) Legend (' admitted ', ' no admitted ', ' decision boundary ') hold off% plot jfigureplot (0:max_ ITR-1, J, ' o--', ' Markerfacecolor ', ' r ', ' Markersize ', 8) xlabel (' Iteration ');  Ylabel (' J ')% Display JJ

Reference documents:

http://openclassroom.stanford.edu/MainFolder/DocumentPage.php?course=DeepLearning&doc=exercises/ex4/ Ex4.html

Http://www.cnblogs.com/tornadomeet/archive/2013/03/16/2963919.html

Deep Learning: Four (logistic regression exercise)