Newton Method-Andrew ng machine Learning public Lesson Note 1.5

Newton method

Reprint Please specify source: http://www.cnblogs.com/BYRans/

In the handout "linear regression, gradient descent," and "logistic regression" we mentioned that θ can be solved by means of gradient descent or gradient rise. Another way to solve θ is explained in this article: Newton's Method (Newton ' s methods).

Newton methods (Newton's method)

sigmoid function g (z) and gradient rise are used to maximize logistic regression? (θ). Now let's talk about another maximization? (θ) algorithm----Newton's method.

The Newton method is to use an iterative method to find the θ value of the f (θ) =0 , where theta is a really worth, not a parameter. The mathematical expression is:

Newton's method is simple to understand that a point is randomly selected and then the tangent of f at that point is calculated, i.e. the derivative of f at that point. The tangent is equal to 0 points, that is, the point where the tangent intersects the x-axis is the value of the next iteration. Until the point where F equals 0 is approximated. Process such as:

Newton method Maximization Likelihood

Newton's method provides a method for finding the θ value of the f (θ) =0. How to maximize the likelihood function ? What is the maximum value of the first derivative at the corresponding point? (θ) to zero. So let f (θ) =? ' (θ), maximized ? (θ) can be converted to: Newton's method of seeking ? (θ) The problem of =0 Theta . The expression of the Newton method, the iterative update formula forθ is:

Newton-Slavic iteration (Newton-raphson method)

in the logistic regression, θ is a vector, so we generalize the above expression to the multidimensional case of the Newton-Slavic Iteration Method (Newton-raphson method), the expression is as follows:

Represented in an expression ? ( θ) The partial derivative of the pair; h is a n*n matrix called the HEssian Matrix . The expression for the Hessian matrix is:

Newton method vs Gradient Descent

As an example of minimizing a goal equation, the red curve is solved using Newton's method, and the green curve is solved by the gradient descent method:

Newton's method usually converges faster than gradient descent and has less iteration frequency.

However, because the inverse of the Hessian matrix is to be computed, the computational amount of each iteration is larger. When the Hessian matrix is not large, the Newton method is better than the gradient descent.

Newton Method-Andrew ng machine Learning public Lesson Note 1.5