On some problems in the optimization

1. Steepest descent method (also called gradient descent method)

The negative gradient direction, the one-dimensional search step, the previous search direction, and the next direction are orthogonal, so it produces jagged images, thus affecting the speed of convergence, especially when X is close to the convergence point.

2. Newton's Method

Use Hesse matrices and gradients to iterate X, resulting in a series of X-Points. The Hesse matrix is required to be non-singular and positive definite, if not, there is no guarantee that the value of the target function decreases and converges to a minimum. If convergence is 2-level convergence, the convergence speed is faster.

3. Resistance damped method

Drag damped method, adding one-dimensional search step, by minimizing the function value, can make the iterative objective function generally decreased.

4. Quasi-Newton method

The above method can not guarantee the Hesse matrix positive definite, so the quasi-Newton method, by constructing the GK positive definite matrix, ensure the Hesse matrix positive definite, and then a one-dimensional search, so it is guaranteed that the function value decline and convergence.

On some problems in the optimization