Selection of Lr_policy in Caffe

When configuring the training network yourself, there are a number of strategies lr_policy This parameter in the Solver file.

Next, take a look at the/caffe-master/src/caffe/proto/caffe.proto file Squadron. Description of this parameter

//The Learning rate decay policy.
The currently implemented learning rate//policies is as follows://-Fixed:always return BASE_LR.  -Step:return BASE_LR * Gamma ^ (floor (Iter/step))//-Exp:return BASE_LR * Gamma ^ iter//-Inv:return       BASE_LR * (1 + gamma * iter) ^ (-power)//-Multistep:similar to step but it allows non uniform steps defined by// Stepvalue//-poly:the effective learning rate follows a polynomial decay, to is//zero by the Max_iter. Return Base_lr (1-iter/max_iter) ^ (power)//-sigmoid:the effective learning rate follows a SIGMOD decay//R Eturn BASE_LR (1/(1 + exp (-gamma * (iter-stepsize)))////Where BASE_LR, Max_iter, Gamma, step, stepvalue and power a


Re defined//in the Solver parameter protocol buffer, and ITER are the current iteration. If you want to see the effect, you can use digits or write your own code display by MATLAB: 

<pre name= "code" class= "plain" >iter=1:50000;
max_iter=50000;
base_lr=0.01;
gamma=0.0001;
power=0.75;
step_size=5000;
%-Fixed:always return BASE_LR.
Lr=base_lr*ones (1,50000); Subplot (2,3,1) plot (LR) title (' fixed ')%-step:return BASE_LR * Gamma ^ (floor (Iter/step)) lr=base_lr. * gamma.^
(iter./10000));
Subplot (2,3,2) plot (LR) title (' Step ')%-exp:return BASE_LR * Gamma ^ iter lr=base_lr * Gamma. ^ iter; Subplot (2,3,3) plot (LR) title (' exp ')%-Inv:return BASE_LR * (1 + gamma * iter) ^ (-Power) lr=base_lr.* (1./(1+gamma.*it
ER). ^power); 
Subplot (2,3,4) plot (LR) title (' Inv ')%-multistep:similar to step but it allows non uniform steps defined by% Stepvalue %-poly:the effective learning rate follows a polynomial decay and to is% zero by the max_iter.
Return Base_lr (1-iter/max_iter) ^ (Power) LR=BASE_LR * (1-iter./max_iter). ^ (power); Subplot (2,3,5) plot (LR) title (' Poly ')%-sigmoid:the effective learning rate follows a sigmod decay% return BASE_LR (1 /(1 + EXP (-gamma * (iter-stepsize))) LR=BASE_LR * (1./(1 + exp (-gamma * (iter-step_size)))); Subplot (2,3,6) plot (LR) title (' Sigmoid ')