Numerical algorithm: One-dimensional search method of unconstrained optimization by dichotomy, Newton and secant method

1. Dichotomy method (first-order guide)

Dichotomy is the method of using the first derivative of the objective function to compress the interval continuously, so in addition to asking F to be a single-peak function, f (x) is continuously differentiable.

(1) Determine the midpoint of the initial interval x (0) = (a0+b0)/2. And then calculates the first derivative F ' (x (0)) at x (0) at f (x), if f ' (x (0)) >0, the minimum point is located on the left side of x (0), that is, the interval of the minimum point is compressed to [A0,x (0)]; Conversely, if F ' (x (0)) <0, the minimum The point is at the right of x (0), the interval of the minimum is compressed to [x (0), B0], and if F ' (x (0) = 0, the description is the minimum point of the function f (x).

(2) construct X (1) According to the new interval, and push it until F ' (X (k)) = 0, stop.

After the N-step iteration, the total compression ratio of the whole interval is (a) n, which is smaller than the total compression ratio of the Golden section method and the Fibonacci sequence method.

2, Newton Method (second-order guide)

It is further requested that f (x) be two consecutive order micro. For a point x (k) on the function f (x), we can use the Taylor formula to construct a polynomial function q (x) =f (x (k)) +f ' (x (k)) (X-x (k)) +1/2 * f ' (x (k)) (X-x (k)) 2, and F (x) in the vicinity of X (k) for local two-time fitting, q (x) can be seen as an approximation of f (x) (near the X (k)), so the minimum point of f (X) can be converted to the minimum of Q (x).

0=q ' (x) =f ' (x (k)) +f ' (x (k)) (X-x (k)) = x = x (k)-F ' (X (k))/F ' (X (k)) so you can choose X (k+1) = x (k)-F ' (X (k)) F ' (X (k))

Newton's method is able to constantly force the first derivative of the objective function f (x) to 0. X (k+1) is generated by a tangent of G (X (k)):

X (k+1) is the tangent of G (X (k)) and the x-axis intersection. With x* X (k), g (x)->0. (g (x) is an approximation of f (x))

Note 1: Newton's method does not need to calculate the function value, but needs to calculate the first derivative and the second order value, and requires F ' (x) >0, if F "(x) <0, then there may be from X (k) to X (k+1) reverse adjustment, convergence to the maximum point, that is, more and more deviate from the minimum.

F ' (x (k)) >0, X (k)->x*

F ' (x (k)) <0, X (k) away from x*

NOTE 2: If the adjustment amplitude of X (k) is too large it may cause the adjustment sequence of x to fluctuate around the minimum point.

The initial point G ' (x (0))/g "(x (0)) is too large to cause the algorithm to fail

(3) Secant method

Newton's method requires the second derivative of f (x), if the second derivative does not exist, it can be approximated by the first derivative of the different points, such as F ' (X (k)) = (f ' (X (k))-F ' (X (k-1)))/(x (k)-X (K-1)), and bring it into the Newton iterative formula, You can get a new iteration formula:

X (k+1) = x (k)-F ' (X (k)) * (X (k)-X (K-1))/(F ' (X (k))-F ' (X (k-1))) <=> x (k+1) = (f ' (X (k)) x (k-1)-F ' (X (k-1)) X (k))/(F ' (X (k))-F ' (X (k-1)))

This method requires two initial points x (-1) and X (0), it can be seen that the secant method does not need to calculate the function value f (x (k)), the second derivative F ' (x (k)), it uses X (k) and X (k-1) between the secant Generation X (k+1):

X (K+1) is produced by a secant of X (k) and X (k-1)

(4) Quasi-parabolic interpolation method

The iterative algorithm is similar to the secant method, and the difference point is the two first derivative of the equation using the function value of three points.

[Test Data]:fx.dat Minx.dat

[Algorithmic code]:test.cpp binaryseg.a binayseg.so newton.a Newton. So secant. A secant.so

Numerical algorithm: One-dimensional search method of unconstrained optimization by dichotomy, Newton and secant method