Comparison of gradient descent method with Newton method in logistic regression model

1. Overview

In the optimization problem of machine learning, the gradient descent method and Newton method are two common methods to find the extremum of convex function, they are all in order to obtain the approximate solution of the objective function. The aim of the gradient descent is to solve the minimum value of the objective function directly, and the Newton's law solves the objective function by solving the parameter value of the first order zero of the objective function in a disguised way. in the parametric solution of logistic regression model, the improved gradient descent method is generally used, and the Newton method can be used.

2 Gradient Descent method2.1 Algorithm Description1, determine the range of errors and the step of descent,determining the function's derivative function

2, while (new value-old value | > Error)

3. Old value = new value

4, new value = initial value-Step * Guide function value, to perform gradient descent

Some problems of the algorithm: the distance between each step is very important near the extremum point, if the steps are too large, it is easy to oscillate near the extremum point and cannot converge. WORKAROUND: Set Alpha to a variable that is decreasing as the number of iterations, but too small can result in many iterations.

2.2. The Java code example code is transferred from: http://www.lailook.net/klxx/04/2016-01-05/51426.html

/** * Gradient descent algorithm, solution f (x) =x^4-3x^3+2 min * derivative is: F ' (x) =4x^3-9x^2 * @author zealot * @date December 13, 2015 */public class Gradientdes cent {//computed, we expect that the local minimum occurs at X=9/4 double x_old = 0;static double x_new = 6;//start Iteration x=6 from Doub Le gamma = 0.01; Step of each iteration double precision = 0.00001;//error static int iter = 0;//iteration number//derivative of the objective function private double  derivative (double x) {Retu RN 4 * MATH.POW (x, 3)-9 *math.pow (x, 2);} private void Getmin () {while (Math.Abs (x_new-x_old) > Precision) {iter++;x_old = x_new;    

3, Newton Method 3.1 algorithm Description

The solution f (x) = 0, if f (x) is conductive, is equivalent to the minimum value of the Iteration x = x-f (x)/F ' (x), the algorithm is as follows:

Input: Initial value x0, error tol; maximum number of iterations m

Input: Approximate solution p or failure information

1, p0=x0

2, while (less than the number of iterations)

3, p= p0-f (p0)/F ' (P0)

4, if |p-p0|<tol, stop, otherwise p0=p

3.2 Examples
4. Reference documents

Comparison of gradient descent method and Newton method in machine learning

Http://www.myexception.cn/cloud/1987100.html

Gradient Descent Method (i) Introduction

http://blog.csdn.net/nupt123456789/article/details/8281923

Java implementation gradient descent algorithm

http://blog.csdn.net/nupt123456789/article/details/8281923

Gradient descent, Newton's method, quasi-Newton method

http://blog.csdn.net/luo86106/article/details/40510383

Comparison of gradient descent method with Newton method in logistic regression model