Optimization algorithm-Conjugate gradient method

 optimization algorithm-Conjugate gradient method(2012-04-20 08:57:05) reprinted

Tags: gossip

the steepest descent method

1. The steepest descent direction

The function f (x) is represented at point X at the rate of change of the number of available party wizards along direction D. For a micro function, the number of square wizards equals the inner product of the gradient and direction, namely:

Df (x;d) =▽f (x) Td,

therefore, the fastest descending direction of function f (x) at point X can be attributed to solving the following nonlinear programming:

min▽f (x) Td

S.T. || d| | ≤1

when d =-▽f (x)/| |   ▽f (x) | |

when the equals sign is established. Therefore, the direction change rate defined at point x along the upper formula is the least, that is, the negative gradient direction is the steepest descent direction.

2. Steepest Descent algorithm

The iterative formula for the steepest descent method is

x (k+1) = x (k) +λkd (k),

where D (k) is the search direction from X (k), which takes the steepest descent direction at x (k), i.e.

d =-▽f (x (k)).

Λk is the step of one-dimensional search from X (k) along direction D (k), i.e. λk satisfies

f (x (k) +λkd (k)) = min F (x (k) +λd (k)) (λ≥0).

The calculation steps are as follows:

(1) Given the initial point x (1) ∈rn, the allowable error ε> 0, set k = 1.

(2) calculate search Direction D =-▽f (x (k)).

(3) if | | D (k) | | ≤ε, the calculation is stopped, otherwise, from X (k), a one-dimensional search is carried out along D (k) for λk, so

f (x (k) +λkd (k)) = min F (x (k) +λd (k)) (λ≥0).

(4) make x (k+1) = x (k) +λkd (k), place k = k + 1, Turn step (2).

  

 

Conjugate Gradient Method

1. Conjugate direction

The core problem of the unconstrained problem optimization method is to select the search direction.

taking the positive definite two function as an example, the geometrical meaning of the conjugate of matrix A is observed in two directions.

There are two functions:

f (x) = (x-x*) TA (x-x*),

where A is a nxn symmetric positive definite matrix, x* is a fixed point, and the equivalent plane of function f (x)

(x-x*) TA (x-x*) = C

is a x*-centric ellipsoidal surface,

▽f (x*) = A (x-x*) = 0,

a positive definite, so x* is the minimum point of f (x).

set X (1) is a point on an equivalent plane, the normal vector at point x (1)

▽f (x (1)) = A (x (1)-x*).

also set D (1) is a tangent vector of this equivalent plane at D (1). Recorded as

d (2) = x*-X (1).

Naturally, D (1) is orthogonal to ▽f (x (1)), i.e. D (1) t▽f (x (1)) = 0, so there are

d (1) TAd (2) = 0,