The convergence rate of the steepest descent method and Newton's method

Open Class at MIT [Introduction to Computer science and programming Using Python] 6.00.1x, Eric Grimson has mentioned that the idea of iterative algorithms is to move the current iteration point in the correct direction to a certain "step", and then verify whether the target value meets certain requirements. At the same time, "direction" and "step size" are the main concerns of different optimization algorithms two aspects. In addition, we also care about the convergence rate of the different optimization algorithms (rates of convergence), which indicates whether the algorithm can approximate the optimal solution in the least number of iterations. Taking the steepest descent method (steep descent) and Newton (Netwon method) as an example, this paper illustrates the convergence rate of different algorithms. This paper mainly refers to the Jorge Nocedal and Stephen J. Wright's numerical Optimization (second edition).

The steepest descent method steep descent Newton Newton method

Steepest descent method (steep descent)

The idea of the steepest descent method is in the opposite direction of the derivative direction of the current iteration point, that is, the fastest descending direction (name origin), which is the direction of the move. One idea of step selection is to select the corresponding step size so that the target function takes the minimum value in the forward direction.

Here we take a simple two-time planning proposition for example, that is f (x) =12xtqx+ctx f (x) = \frac{1}{2}x^t Qx + c^t x where q q is a symmetric positive definite array. For the target function f (x) f (x), the current iteration point is XK x_k, and the first derivative of f (x) f (x) at XK X_k is ∇fk=qx+c \nabla f_k=qx+c. So in the XK x_k place direction is −∇FK-\nabla f_k. To determine the stride length, the following is the minimum value for the optimization proposition with Α\alpha as the decision variable: F (X−Α∇FK) =12 (X−Α∇FK) TQ (X−Α∇FK) +ct (X−Α∇FK) F (x-\alpha \nabla f_k) = \frac{1 }{2} (X-\alpha \nabla f_k) ^t Q (X-\alpha \nabla f_k) + c^t (x-\alpha \nabla f_k) Order the derivative of Α\alpha to 0 0 to obtain the corresponding step size: αk=∇f