Optimal Learning Rate-2-Newton method and Quasi-Newton method

This chapter summarizes the knowledge of optimizing the learning rate, and the pre-knowledge is "linear regression, gradient descent algorithm", so if this chapter you look at the foggy even the learning rate is what you do not know, you need to first pre-knowledge to get it done.

Other Notes

Because the pre-knowledge of this summary is "linear regression, gradient descent algorithm", then the content is "to seek the minimum value of the objective function f (x)" for the purpose.

But don't worry about finding the maxima, because adding a minus sign to F (x) directly converts the problem to a minimum problem.

In the way, personal feeling is because of the research so much to find the smallest things, so everyone for the sake of convenience, usually encountered to find the greatest value of the problem first turned it into the minimum value, so will often see this sentence "Custom we will convert it to the minimum value" ....

Newton's Method

First ask a question: if the search direction is not gradient direction, can it? That is, is it possible to use a different direction to find the minimum value of a function?

The answer is yes, as shown in:

I fit a two-time curve at the XK point and then find the xk+1 = xk+dk point after finding the extremum for the two curve, so we find a smaller value F (xk+1) than F (XK), and then iterate over the process to find the minimum value.

and the direction of this two-time curve is Newton's direction , and this method is Newton 's method.

But because Newton's law could produce the problems shown,

The xk+1 = Xk+dk "ran" after fitting a two-time curve in the XK, which could not make the function converge, and some other reasons, the quasi-Newton method is generally not used in practical work.

But for the record, Newton's method of Xk+1 is still posted here:

xk+1= xk-f ' (XK)/F ' (XK) type 1

The origin of this formula is as follows:

Use the Taylor formula to expand F (x) two times (assuming, of course, the second-order continuity of F (x)):

φ (x) = f (xk) + F ' (XK) (X-XK) + (a) f ' (XK) (X-XK) 2 + R2 (x)

Discard Gorkiyu type after derivation:

φ ' (x) = F ' (XK) + F ' (XK) (X-XK)

Another φ ' (x) = 0:

xk+1= xk-f ' (XK)/F ' (XK)

PS1: To extend the above formula to multi-dimensional method is the "use of the number of Square wizard to replace the first-order guide, with the Hessian matrix instead of the second derivative", that is: Xk+1 = xk-h-1▽f (XK), where xk and xk+1 are vectors, H-1 represents the inverse of the Hessian matrix.

PS2: Why the formula 1 can correspond to the Newton direction of the curve is not very clear, if you know whether you can tell me in the comments, or send a link, or tell me to check what knowledge points are good, greatly appreciated.

Quasi-Newton

As shown, the disadvantage of Newton's method is that we can fall along the gradient direction in the XK and find the extremum point (this is the first order, only one-step guide), but along the second-order convergence (there are second-level guide, Why?). And then look at the Newton method of the formula) in the direction to optimize the instead of rising up. That is: the first order can be convergent, can be the second-level may not converge, so it is thinking: can make a step lower than a little faster, but the efficiency of less than the second-degree Newton method, but the real can be convergent method? So he made a direction like Newton, which has the quasi-Newton method.

In order to facilitate the explanation, here is the formula of the Newton method is affixed:

xk+1= xk-h-1▽f (XK), where XK and xk+1 are both vectors

The quasi-Newton is to replace the inverse matrix of the Hessian matrix or the Hessian matrix with an approximate matrix.

The quasi-Newton algorithms commonly used in practice are:

Dfp:davidon-fletcher-powell

Bfgs:broyden-fletcher-goldfarb-shanno

Dfp

Let's look at the following two points of knowledge:

One-step descent: xk+1 = xk-α▽f ' (XK)

Hige method: xk+1 = xk-α▽f ' (XK)/▽f ' (XK)

Then begin to explain the DFP derivation process.

PS: The following θ is equivalent to the above x, that is, before the function f (x), the following is said function f (θ), it is not special meaning, is a need ....

The gradient of the function f (θ) is g (θ) and the second-order guide function is h (θ).

PS: If θ is an n-dimensional vector, then g (θ) is also an n-dimensional vector, and H (θ) is a n*n matrix.

Next, as with the use of Taylor to find the Newton method of the formula, using the Taylor expansion to derive DFP, thus the following derivation process:

Explain:

First line: Taylor unfold

Second line: Discard Gorkiyu-type get

The third line: derivation, where the second line of the last part of the derivation takes advantage of this knowledge point: When matrix A is a symmetric array (a = at), XT The result of a x to X is: 2a x, and the Hession matrix is symmetric array.

Line four: Since this formula to θi set up, that to θi-1 also must set up

Line five: the G (θi) abbreviated to GI,G (ΘI-1) abbreviated into GI-1

Line six to line seventh: Use the formula in the lower-right corner of the graph to replace the sixth line.

The result is the last formula above:

Ci GI = θi

That

Variation of Inverse * gradient of hession matrix = Variation of independent variable

Now we know, our goal is to calculate CI, because:

For Newton's method: Xk+1 = Xk-h-1▽f (XK), if you can find H-1, then this equation is completely done.

And this is also the same Ah, the most owe ci ....

How can we ask for CI? The method is to do the following approximate calculation of CI, that is: CI May and Ci-1 a little relationship, namely:

ci= Ci-1 + C.

Because CI is a hession matrix, C also needs to be a phalanx, so another:

C =av vt,u is a column vector

That

ci= Ci-1 + av vt

In practice, however, it is found that Ci and Ci-1 + av vt are too approximate, so the above formula is modified to:

ci= Ci-1 +ai-1v i-1 v i-1t+ b i-1ui-1 u i-1t style 2

All right, now there are two of these formulas:

To put the second descendant of the above into the first formula is:

Ci-1 gi + ai-1vi-1vi-1t gi + b i-1u i-1 u i-1t gi = θi type 3

Now look at the above equation, which is known to have:

Θi (the current argument-the last argument)

GI (Current gradient-previous gradient)

Ci-1 (last used Ci-1)

Unknown: α, V, beta, u

And because, if the formula 3

Ci-1 gi =-B i-1u i-1 u i-1t gi

ai-1vi-1vi-1t GI = θi

, the equal sign of equation 3 is satisfied.

Then define:

Ui-1 = Ci-1 Gi①

b i-1u i-1t gi =-1

vi-1= Θi②

ai-1vi-1t GI = 1

So as to find out

ai-1= 1/(vi-1t gi) ③

Bi-1 = -1/(u i-1t gi) ④

The iterative formula of the DFP algorithm is obtained after substituting the ①②③④ in 2 :

- 4

So do you remember the formula of Newton's law? That

xk+1= xk-h-1▽f (XK)

Now, using DFP to find a matrix ci that can be substituted for H-1, use CI instead of H-1 to get the quasi-Newton formula:

xk+1 = Xk-ci▽f (XK)

However, the actual use of ci▽f (XK) is often added to a very small parameter α, such as α=0.01, thus forming a quasi-Newton with damping , namely:

xk+1 = xk-αci▽f (XK)

The code for the iterative formula of the DFP algorithm (from Shambo teacher of the Little Elephant Academy)

A and B are formula 4 The right side of the equals sign, the second and third items

Bfgs

Mathematicians used DFP A few years later to find out how to think:

For the DFP algorithm, we already know it by the following formula

subgroups 1

To obtain the final result

- 5

If you multiply both sides of the first expression of the subgroup 1 by the Ci-1, you get:

GI = Hi θi

PS: Because CI =hi-1

subgroups The second expression of 1 makes the same thought, so that the hession matrix equals such an iteration:

hi= Hi-1 +ai-1v i-1 v i-1t+ b i-1ui-1 u i-1t

That is, there are subgroups

In the same vein, we can draw the following conclusion:

- 6

Compared with the following formula 5 and 6, you can see:

Equation 5 is the inverse matrix of the hession matrix

Formula 6 is Hession matrix

In that case, is there any way to directly seek the hession? Yes, the answer is the BFGS algorithm.

PS:DFP algorithm CI is also hession inverse, but DFP algorithm will produce some problems, so along the direction of DFP research BFGS algorithm, like IPv4 and IPv6 relationship.

The following goal is to find the inverse of the formula 6 .

Here we use Sherman-morrisong formula, namely: If A is n-order invertible matrix, u, V is n Willi Vector, if vta-1u≠-1, then

- 7

And then:

1, the first two on the right of the equation 6 equals to a

2, formula 6 hi-1 θi as X, the third denominator as a (the result of the denominator is the number), the third is (x/a) * XT, and then x/a as u,x as V

3, so that the formula 6 corresponds to the equation 7 , at this time to use the Sherman-morrisong equation

4, in the equation 6 equals to the right of the first two Hi-1 as a, gi/(GiT θi) as u, gi as a v after the use of a sherman-morrisong formula, and finally get the following results:

This is the formula for the BFGS algorithm .

Code (Shambo teacher from the Academy of small Elephants)

Use the method, such as the following logistics regression:

C in the code is the CI in the formula of the BFGS algorithm , if C is the unit array that quasi-Newton degenerate in order to decline the gradient, the reverse is quasi-Newton, that is, for w = w + Alpha * C.DOT (g), if C is a unit array, then the C.dot (g) is the G itself, If c is not a unit array, then it is better to use Hessian_r to find the inverse matrix of hession.

At last

One of the L-BFGS is interested in searching by himself, which I haven't studied yet.

Optimal Learning Rate-2-Newton method and Quasi-Newton method