Deep Learning Notes: Summary of Optimization methods (Bgd,sgd,momentum,adagrad,rmsprop,adam)

Source: Internet
Author: User

from:http://blog.csdn.net/u014595019/article/details/52989301

Recently looking at Google's deep learning book, see the Optimization method that part, just before with TensorFlow is also to those optimization method smattering, so after reading on the decentralized, mainly the first-order gradient method, including SGD, Momentum, Nesterov Momentum, Adagrad, Rmsprop, Adam. Where Sgd,momentum,nesterov Momentum is manually specified for the learning rate, and the latter Adagrad, Rmsprop, Adam, can automatically adjust the learning rate.
Second-order method at present my level is too poor, can not understand .... don't put it up. BGD

That is, batch gradient descent. In training, every step of the iteration uses all the content of the training set. That is to say, using the existing parameters to generate an estimated output yi^ for each input in the training set, and then compare with the actual output Yi, statistics all errors, averaging after the average error, as a basis for updating parameters.

Specific implementation:
Required: Learning rate Ε, initial parameter θ
Each step of the iteration process:
1. Extract all contents of the training set {x1,..., xn}, and related output yi
2. Calculate gradients and errors and update parameters:
g^←+1n∇θ

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.