Re-discovering Gradient descent method--backtracking line Search

Always thought the gradient dropped very simple, the result recently found that I wrote a gradient drop particularly slow, and then finally found the reason: the choice of step size is very important, there is a gradient descent method called backtracking line search is very efficient, the algorithm described in:

Here is a simple example to show an unconstrained optimization problem:

Minimize y = (x-3) * (x-3)

Here is the Python code, which compares two methods

#-*-coding:cp936-*-#optimization Test, y = (x-3) ^2 fromMatplotlib.pyplotImportFigure , hold, plot, show, Xlabel, Ylabel, Legenddeff (x):"The function we want to minimize"        return(x-3) **2defF_grad (x):"gradient of function f"        return(x-3) x=0y=f (x) Err= 1.0Maxiter= 300Curve=[Y]it=0step= 0.1#here is the method I used earlier, it seems quite reasonable, but very slow whileErr > 1e-4 andIt <Maxiter:it+ = 1Gradient=F_grad (x) new_x= X-gradient *Step new_y=f (new_x) New_err= ABS (New_y-y)ifNew_y > Y:#If there are signs of divergence, reduce step sizeStep *= 0.8err, x, y=New_err, new_x, new_yPrint 'ERR:'Err', y:', y curve.append (y)Print 'iterations:', Itfigure (); hold (True); Plot (Curve,'r*-') Xlabel ('iterations'); Ylabel ('Objective function Value')#Backtracking line Search is shown below, fastx =0y=f (x) Err= 1.0Alpha= 0.25Beta= 0.8Curve2=[Y]it=0 whileErr > 1e-4 andIt <Maxiter:it+ = 1Gradient=F_grad (x) Step= 1.0 whileF (x-step * gradient) > Y-alpha * Step * gradient**2: Step*=Beta x= X-step *Gradient new_y=f (x) Err= y-new_y y=new_yPrint 'ERR:'Err', y:', y curve2.append (y)Print 'iterations:', Itplot (Curve2,'bo-') Legend (['gradient descent I used','Backtracking Line Search']) show ()

Running results such as:

What is better, at a glance

My method used 25 iterations, and backtracking line search was used only 6 times. (and the method I used before does not necessarily converge, for example, if you change the first method of the Stepsize to 1, you will find that no convergence to the optimal solution to stop, this is a bug, to note)

It's just a toy example, and the efficiency difference between the two is more pronounced in my real use of the optimization problem, which is estimated to be 10 times times as much.

--

From the article: Https://www.youtube.com/watch?v=nvZF-t2ltSM

(It is the optimization course of CMU)

Re-discovering Gradient descent method--backtracking line Search