First, Newton Lawin the blog "Optimization Algorithm-Newton (Newton method)" introduced the Newton method of thinking, Newton's method has a second-order convergence, compared to the steepest descent method, convergence faster. In Newton's method, information about the second derivative of the function is used, for the function, which represents the vector. In the process of solving the Newton method, the first is to expand the function at the place, the expansion type is:
wherein, the object is represented by the gradient of the function, is a vector. , which indicates that the target function is in theHesseMatrix. Omit the last high-order infinity, which is the following:
on both sides of the derivative, that is:
in the basic Newton method, the leading value at the point where the maximum is obtained is the left side of the upper type. Then:
to find out:
found in the above-described, in the Newton method requiredHesseThe matrix is reversible. At the time, the formula was:
at this point, whether it can be passed,, and simulated outHessethe construction process of the matrix? This method is called quasi-Newton method (Quasinewton), the upper type is called quasi-Newton equation. In quasi-Newton method, we mainly include DFP Quasi-Newton method andBFGS Quasi-Newton method . Two,DFP Quasi-Newton method 1.Introduction toDFP Quasi-Newton method DFP Quasi-Newton method is also called DFP correction method,DFP Correction method is the first quasi-Newton method, is a Davidon It was first proposed, followed by Fletcher and the Powell explained and improved, named with the initials of the three-person name. for the quasi-Newton equation:
Simplification can be done by:
order, you can get:
In the DFP correction method, assume that:
2 . Derivation of DFP correction method Order: The vectors in which all are. ，。 the quasi-Newton equation can be simplified to:
will be put into the formula:
will be put into the formula:
known: As a real number, as a vector. In the formula, there are many possibilities of the parameter reconciliation, we take the special case, suppose,. Then:
on-behalf:
order, then:
then the finalDFPThe correction formula is:
3, solve the specific optimization problemsSolving unconstrained optimization problems
among them,.
pythonProgram implementation:
- function.py
#coding: UTF-8 "Created on May 19, 2015 @author:zhaozhiyong" from numpy Import * #fundef fun (x): return * (x[0,0] * * 2-x[1,0]) * * 2 + (x[0,0]-1) * * 2#gfundef gfun (x): result = Zeros ((2, 1)) result[0, 0] = x * x[0,0] * (x[0, 0] * * 2-x[1,0]) + 2 * (x[0,0]-1) result[1, 0] = -200 * (x[0,0] * * 2-x[1,0]) return result
- dfp.py
#coding: UTF-8 "Created on May 19, 2015 @author:zhaozhiyong" from NumPy import *from function Import *def DFP (fun, Gfun, x0 ): result = [] Maxk = 0.55 Sigma = 0.4 m = shape (x0) [0] Hk = Eye (m) k = 0 while (K < MAXK): GK = Mat (Gfun (x0)) #计算梯度 DK =-mat (Hk) *gk m = 0 mk = 0 while (M < 20): Newf = Fun (x0 + rho * * M * dk) oldf = Fun (x0) if (Newf < Oldf + Sigma * (RHO * m) * (Gk. T * DK) [0,0]): mk = m break m = m + 1 #DFP校正 x = x0 + Rho * * mk * DK sk = x-x0 YK = gfun (x)-GK if (SK. T * YK > 0): HK = HK-(HK * YK * yk). T * Hk)/(YK. T * Hk * yk) + (SK * sk. T)/(SK. T * yk) k = k + 1 x0 = x result.append (fun (x0)) return result
- testdfp.py
#coding: UTF-8 "Created on May 19, 2015 @author:zhaozhiyong" from BFGS import *from DFP import Dfpimport Matplotlib.pyplot as plt x0 = Mat ([[[ -1.2], [1]]) result = DFP (fun, Gfun, x0) n = len (result) ax = Plt.figure (). Add_subplo T (111) x = arange (0, N, 1) y = Resultax.plot (x, y) plt.show ()
4. Experimental results
Optimization algorithm--DFP algorithm of quasi-Newton method
