Exercise: Logistic regression and Newton's method

Question address:

Exercise: Logistic Regression

Question summary:In a high school, there are 80 students, 40 of whom are admitted to the university, and 40 are not. X contains the scores of 80 students in two standard examinations, and y includes whether the students are admitted (1 indicates admission, 0 indicates not admission ).

Process:

1. Load Test DataAnd add an offset for the X input.

 
X = load ('Ex4x. dat'); Y= Load ('Ex4y. dat'); X= [Ones (length (Y ),1) X];
 

2. Draw Data Distribution

  % find returns the indices of the %    rows meeting the specified condition  POS  = find (y =  1 ); neg = find (y =  0  );   % assume the features are in the 2nd and 3rd %    Columns of x  plot (x (Pos,   2  ), X (Pos,  3 ),  '  +   ' ); hold onplot (x (neg,   2 ), x (neg,  3  ),  '  O   ') 

3. Newton's method

First, let's recall the assumption of Logistic regression:

Because there is no Sigmoid Function in MATLAB, we use niline to define one:

 
G = inline ('1.0./(1.0 + exp (-z ))');% Usage: to find the value of the sigmoid % evaluated at 2, call G (2)
 

Let's take a look at the defined cost function J (θ ):

We want to use the Newton's method to obtain the minimum value of Cost Function J (θ. Recall that the iteration rule of θ in Newton's method is:

In logistic regression, the gradient and Hessian methods are as follows:

Note that the preceding formula is written in Vector Form.

Where, the vector is n + 1*1, which is a matrix of N + 1 * n + 1.

And scalar.

Implementation

Follow the methods described in the Newton's method described above to gradually implement.CodeAs follows:

Function [Theta, J] = Newton (x, y)  % Newton summary of this function goes here %  Detailed explanation goes hereM = Length (y); Theta = Zeros ( 3 , 1  ); G = Inline ( '  1.0./(1.0 + exp (-z ))  ' ); POS = Find (y = 1  ); Neg = Find (y = 0  ); J = Zeros ( 10 , 1  );  For Num_iterations = 1 : 10      %  Calculate the actual outputH_theta_x = G (x * Theta ); % Convert y = 0 and Y =  When 1 is calculated separately and then added, the J function is calculated.Pos_j_theta =- 1 * Log (h_theta_x (POS); neg_j_theta =- 1 * Log (( 1 - H_theta_x (NEG); j (num_iterations) = Sum ([pos_j_theta; neg_j_theta])/ M;  %  Calculate the J derivative and Hessian MatrixDelta_j = Sum (repmat (h_theta_x-y ), 1 ,3 ).* X); H = X '  * (Repmat (h_theta_x. * (1-h_theta_x), 1, 3). * X );      %  Update θTheta = Theta-inv (h) * delta_j '  ;  End  %  Now plot J % Technically, the first J starts at the zero- ETH Iteration % But Matlab/octave doesn ' T have a zero Index  Figure; plot (  0 : 9 , J ( 1 : 10 ), '  -  '  ) Xlabel (  '  Number of iterations  '  ) Ylabel (  '  Cost J '  ) End 

PS: the code for directly calculating the J function given in the answer to the exercise appendix is very elegant:

 
J (I) = (1/m) * sum (-y. * log (h)-(1-y). * log (1-H ));
 

Call in MATLAB:

[Theta J] = Newton (x, y );

The output is as follows:

θ:

The output image of the J function is (when the number of iterations is 10 ):

As you can see, the iteration has actually converged to the fourth time.

We can output the line between admitted and unadmitted. The Code is as follows:

Plot (x (Pos, 2 ), X (Pos, 3 ), '  +  '  ); Hold onplot (x (neg, 2 ), X (neg, 3 ), '  O  '  ) Ylabel (  '  Exam 2 scores  '  ) Xlabel (  '  Exam 1 scores  '  ) Plot (x (:,  2 ), (Theta ( 1 )-X (:,2 ) * Theta ( 2 )/Theta ( 3 ), '  -  '  ); Legend (  '  Admitted  ' , '  Unadmitted  ' , '  Demo-boundary  ' );

Effect