How to select ADAM,SGD Neural network optimization algorithm

Before on the TensorFlow and Caffe on the CNN used to do video processing, in the study of TensorFlow example, the code gives the optimization scheme by default in many cases is directly using the Adamoptimizer optimization algorithm, as follows:

optimizer = tf.train.AdamOptimizer(learning_rate=lr).minimize(cost)

• 1

However, in the use of Caffe solver inside the general use of Sgd+momentum, as follows:

base_lr: 0.0001momentum: 0.9weight_decay: 0.0005lr_policy: "step"

• 1
• 2
• 3
• 4

Plus recently read an article: The marginal Value of Adaptive Gradient Methods
In the machine learning article link, the paper also discusses the adaptive optimization algorithm: Adagrad, Rmsprop, and Adam and SGD algorithm performance between the comparison and selection, so move a conclusion and feelings.

Abstract

After the experiment, the most important conclusion is:

that the solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when these solutions have better training performance. Theseresults suggest that practitioners should reconsider the use of adaptive methods to train neuralnetworks

• 1
• 2
• 3

The adaptive optimization algorithm usually results in a worse performance than the SGD algorithm (often a lot worse), although the Adaptive optimization algorithm performs well during training, so users need to think carefully when using an adaptive optimization algorithm! (finally know why CVPR's paper all used SGD instead of the most diao Adam in theory)

Introduction

The author continues to give the dry goods conclusion:
Our experiments reveal three primary findings.

First,WithThe same amountof Hyperparameter tuning, SGDand SGDWith Momentum outperformadaptive methodsOnThe Development/testSet across all evaluated modelsand tasks. ThisIsTrue even whenThe adaptive methods Achievethe same training loss or lower than non-adaptivemethods. Second, adaptive methods often display faster initial progress on the training set, buttheir performance quickly plateaus on the development/test set. Third, the same amount of Tuningwas required for all methods, including adaptive methods. This challenges the conventional Wisdomthat Adaptive Methods Require less tuning. 

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8

Translation:
1: With the same number of parameters to the parameter, SGD and SGD +momentum The amount of the performance on the test set is better than all the adaptive optimization algorithms, although sometimes the adaptive optimization algorithm loss smaller on the training set, but their loss on the test set is still higher than the SGD method,
2: The adaptive optimization algorithm converges faster in the training set in the pre-training stage, but this is a bit of a bottleneck in the test set.
3: All methods require the same number of iterations, which is contrary to the conclusion that the conventional default adaptive optimization algorithm requires fewer iterations!

Conclusion

Post the results of several experiments by the author:

You can see that SGD is not the fastest loss in pre-training, but the perplexity confusion on test set (where the link content is written) is minimal.

Using the SGD algorithm in TensorFlow: (Reference)

 # Global_step training_iters= Len (Data_config[ ' Train_label ']) global_step=training_iters*model_config[ ' N_epoch '] decay_steps=training_iters*1 #global_step = tf. Variable (0, name =  ' Global_step ', Trainable=false" Lr=tf.train.exponential_decay (Learning_rate=model_config[ Learning_rate '], Global_step=global_step, decay_steps=decay_steps, Decay_rate=0.1, Staircase=false, Name=none) optimizer= Tf.train.GradientDescentOptimizer (LR). Minimize (Cost,var_list=network.all_params)     

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced. 72867109

How to select ADAM,SGD Neural network optimization algorithm