The Newton method of machine learning

Taylor Formula

First look at the Taylor formula, for the function, if the function is smooth and a point exists in the order of the derivative, you can use a polynomial to describe the approximate value of the point neighborhood. The formula is as follows:

Newton's Method

Newton's method is generally used to solve the root of the equation and solve the extremum.

In addition to the gradient descent method, the numerical optimization algorithm has a common method of Newton method. For nonlinear equations, the Newton iterative method can be used to solve the problem, and it converges fast.

The basic idea is: for a nonlinear function f (x), according to the Taylor formula to get a point near x xk x_{k} Expansion of the polynomial can be used to approximate the value of the function f (x), the polynomial corresponding to the function f (x), the minimum value of f (x) as a new iteration point, and then continue at the new iteration point Taylor Formula expansion, Until the minimum value is obtained to meet certain accuracy. principle

Assuming that the function f (x) two times can be slightly, then two times Taylor expands,
F (x) ≈g (x) =f (XK) +f′ (XK) (X−XK) +12f′′ (XK) (X−XK) 2 F \left (x \right) \approx g \left (x \right) = f \left (x_k \right) +{ F} ' \left (X_k \right) \left (X-x_k \right) +\frac{1}{2}{f} ' \left (X_k \right) \left (x-x_k \right) ^2
g (x) polynomial is the approximation of f (x), the function f (x) Extremum can be converted to a derivative function of 0, a derivative of g (x) and make it 0,
f′ (XK) +f′′ (XK) (X−XK) =0 F ' \left (X_k \right) +{f} ' \left (X_k \right) \left (x-x_k \right) =0
Get
x=xk−f′ (XK) f′′ (XK) x=x_k-\frac{{f} ' \left (X_k \right)}{{f} ' \left (X_k \right)}
The iterative formula is obtained,
xk+1=xk−f′ (XK) f′′ (xk