Summary of deep Learning optimization method

http://blog.csdn.net/lien0906/article/details/47399823 excerpt from this blog
In August 15, the Adam method was added to the Caffe.
Stochastic Gradient descent (SGD) Parameters for SGD

When using a learning method with random gradient descent (SGD), there are generally the following parameters to adjust: Learning rate Weight Decay weight decay Momentum momentum Learning rate Decay learning rates decay

In this case only the first parameter (learning rate) is necessary, the remainder is to improve the adaptive parameters, that is, the latter 3 parameters can be set to 0 when not required. Learning Rate

The learning rate determines the speed at which the weights are updated, so that the results pass through the optimal value, too small, which slows down too slowly. Only by human intervention to adjust the parameters need to constantly modify the learning rate, so the following 3 parameters are based on the self-adaptive approach proposed solution.
Wi←wi−η∂e∂wi

Weight Decay

In the practical application, in order to avoid the over-fitting of the model, we need to add the cost function to the specification, and in SGD we add $−ηλw_i$ to normalize the cost function.
Wi←wi−η∂e∂wi−ηλwi
The basic idea of this formula is to reduce the effect of unimportant parameters on the results, and the useful weights will not be affected by weight decay, which is very similar to dropout's thought principle.

Link 1
Link 2 Learning rate Decay

A way to improve the ability of SGD optimization by decreasing the size of the learning rate for each iteration. Initial learning rate $\eta=\eta_0$ learning, decay $\eta_d$ at each iteration $s $:

η (s) =η01+s⋅ηd
In many papers, another common method is to iterate around 30-50 times to directly manipulate the learning rate ($\eta←0.5\cdot\eta$)