How to choose adam,sgd__ Algorithm for neural network optimization algorithm

Before the TensorFlow and Caffe on the CNN used to do video processing, in the learning TensorFlow examples of the code to give the optimization scheme by default in many cases are directly used Adamoptimizer optimization algorithm, as follows:

Optimizer = Tf.train.AdamOptimizer (LEARNING_RATE=LR). Minimize (Cost)

But in the use of Caffe when the solver is generally used in Sgd+momentum, as follows:

base_lr:0.0001
momentum:0.9
weight_decay:0.0005
lr_policy: "Step"

Plus recently read an article: the marginal Value of adaptive gradient Methods
In Machine learning article link, this paper also explores the comparison and selection between adaptive optimization algorithms: Adagrad, Rmsprop, and Adam, and the SGD algorithm, so move on to a conclusion and a feeling. Abstract

After the experiment of this paper, the most important conclusion is:

We observe that's solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when t Hese solutions have better training performance. These
results suggest this practitioners should reconsider the use of adaptive methods to train neural

The adaptive optimization algorithm usually gets the result of worse (often worse) performance than the SGD algorithm, although the Adaptive optimization algorithm performs well in training, so the user needs to consider it carefully when using adaptive optimization algorithm. (I finally know why CVPR's paper are all in SGD, not the most diao Adam in theory) Introduction

The author continued to give dry conclusion:
Our experiments reveal three primary findings.

The
same amount of hyperparameter tuning, SGD and SGD with momentum outperform adaptive methods on
the Development/test set across all evaluated models and tasks. This is
true even when the adaptive methods achieve the same training loss or lower than non-adaptive
. Second, adaptive methods often display faster initial progress on the training set, but their performance quickly, Plat
Eaus on the development/test set. Third, the same amount of tuning
was required to all methods, including adaptive. This challenges the conventional wisdom

Translation:
1: Using the same number of parameters to tune the parameter, SGD and SGD +momentum methods The frontal error on the test set is better than all the adaptive optimization algorithms, although sometimes the loss of adaptive optimization algorithms is even smaller in the training set, but their loss on the test set is still higher than the SGD method.
2: The adaptive optimization algorithm converges faster in the training set at the early stage of training, but this kind of bottleneck is encountered in the test set.
3: All methods require the same number of iterations, which contradicts the conclusion that the conventional default adaptive optimization algorithm requires fewer iterations. Conclusion

Paste some of the experimental results of the author's work:

You can see that SGD is not the fastest loss in the early training period, but the perplexity confusion (write link content) on test set is minimal.

Using the SGD algorithm in TensorFlow: (Reference)

    # global_step
    training_iters=len (data_config[' Train_label '])
    global_step=training_iters*model_config[' n _epoch ']
    decay_steps=training_iters*1
    #global_step = tf. Variable (0, name = ' Global_step ', Trainable=false)
    Lr=tf.train.exponential_decay (learning_rate=model_config[' Learning_rate '],
        global_step=global_step, Decay_steps=decay_steps, decay_rate=0.1, Staircase=false, Name=None)

 Optimizer=tf.train.gradientdescentoptimizer (LR). Minimize (Cost,var_list=network.all_params)