Method of solving unconstrained optimization problem

Most commonly used: Gradient descent method (gradient descent) or steepest descent method (steepest descent). Have the advantages of simple implementation. Gradient descent is an iterative algorithm, and each step needs to solve the gradient of the objective function.

When the objective function is a convex function, the gradient descent algorithm is the global optimal solution. Under normal circumstances, the solution is not guaranteed to be the global optimal solution. The convergence rate of gradient descent method is not necessarily the fastest.

Also, Newton's method and quasi-Newton method are common methods, which have the advantages of fast convergence speed. Newton algorithm is an iterative algorithm, each step needs to solve the objective function of the sea-race matrix, calculation is responsible for the comparison.

The quasi-Newton algorithm simplifies the calculation process by using the inverse matrix or the sea-race matrix of the positive definite matrix approximation of the sea-race matrix. The improved algorithm of "simulating" Newton method. The basic idea is to construct a positive definite symmetric array which can approximate the inverse of the sea-race matrix without the second derivative, thus optimizing the objective function under the condition of "quasi-Newton". The different tectonic methods of the sea-race determine the different quasi-Newton methods.

Other methods: DFP (Davidon-fletcher-powell) algorithm, BFGS (Broyden-fletcher-goldfarb-shanno) algorithm, Broyden class algorithm. These methods are actually quasi-Newton methods.

Newton's Law related blog: http://jacoxu.com/?p=146

Quasi-Newton method related blog: http://www.tuicool.com/articles/EviQ32m http://www.codelast.com/?p=2780

Supplementary information: Simplex method, one of the ten most awesome algorithms of the 20th century, still not understand, post a Wikipedia link: https://en.wikipedia.org/wiki/Simplex_algorithm

Method of solving unconstrained optimization problem