Newton Method-Andrew ng machine Learning public Lesson Note 1.5

Source: Internet
Author: User
Tags andrew ng machine learning

Newton method

Reprint Please specify source:

In the handout "linear regression, gradient descent," and "logistic regression" we mentioned that θ can be solved by means of gradient descent or gradient rise. Another way to solve θ is explained in this article: Newton's Method (Newton ' s methods).

Newton methods (Newton's method)

sigmoid function g (z) and gradient rise are used to maximize logistic regression? (θ). Now let's talk about another maximization? (θ) algorithm----Newton's method.

The Newton method is to use an iterative method to find the θ value of the f (θ) =0 , where theta is a really worth, not a parameter. The mathematical expression is:

Newton's method is simple to understand that a point is randomly selected and then the tangent of f at that point is calculated, i.e. the derivative of f at that point. The tangent is equal to 0 points, that is, the point where the tangent intersects the x-axis is the value of the next iteration. Until the point where F equals 0 is approximated. Process such as:

Newton method Maximization Likelihood

Newton's method provides a method for finding the θ value of the f (θ) =0. How to maximize the likelihood function ? What is the maximum value of the first derivative at the corresponding point? (θ) to zero. So let f (θ) =? ' (θ), maximized ? (θ) can be converted to: Newton's method of seeking ? (θ) The problem of =0 Theta . The expression of the Newton method, the iterative update formula forθ is:

Newton-Slavic iteration (Newton-raphson method)

in the logistic regression, θ is a vector, so we generalize the above expression to the multidimensional case of the Newton-Slavic Iteration Method (Newton-raphson method), the expression is as follows:

Represented in an expression ? ( θ) The partial derivative of the pair; h is a n*n matrix called the HEssian Matrix . The expression for the Hessian matrix is:

Newton method vs Gradient Descent

As an example of minimizing a goal equation, the red curve is solved using Newton's method, and the green curve is solved by the gradient descent method:

Newton's method usually converges faster than gradient descent and has less iteration frequency.

However, because the inverse of the Hessian matrix is to be computed, the computational amount of each iteration is larger. When the Hessian matrix is not large, the Newton method is better than the gradient descent.

Newton Method-Andrew ng machine Learning public Lesson Note 1.5

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.