An introduction to optimization algorithms in the most complete machine learning

In machine learning, there are many problems, there is no analytic form of solution, or analytic form of the solution but the computation is very large (for example, the problem of the least-squares solution), for such problems, we usually choose to use an iterative optimization method to solve.
These commonly used optimization algorithms include gradient descent method (gradient descent), conjugate gradient method (conjugate gradient), momentum algorithm and its variants, Newton and Quasi-Newton methods (including L-BFGS), Adagrad,adadelta , Rmsprop,adam and its variants, Nadam. gradient Descent method (gradient descent)

Imagine you are on a mountain, without considering other factors, how you can walk to the bottom of the hill as fast as possible. Of course it is the most steep place to choose, and this is the core idea of the gradient descent method: it gradually approximates the minimum value of a function by "stepping" forward each time in the current gradient direction (the steepest direction).
In nth n iterations, the parameter Θn=θn−1+δθ\theta_n=\theta_{n-1}+\delta\theta
We perform a first-order Taylor expansion of the loss function at θn−1 \theta_{n-1}:
L (θn) =l (θn−1+δθ) ≈l (θn−1) +l′ (θn−1) δθl (\theta_n) =l (\theta_{n-1}+\delta\theta) \approx L (\theta_{n-1}) +L^{'} (\theta _{n-1}) \delta\theta
In order to make L (θn) <l (θn−1) L (\theta_n) \lt L (\theta_{n-1}), it is advisable δθ=−αl′ θn−1 (\delta\theta=-\alpha) l^{'} (\theta_{n-1}), That's the iterative formula that gets our gradient descent:
θn:=θn−1−αl′ (θn−1) \theta_n:=\theta_{n-1}-\alpha l^{'} (\theta_{n-1})

The gradient descent method, based on the number of samples that are brought in by the L l of each solution loss function, can be divided into: total gradient Descent (calculate all sample losses) , batch gradient drop (loss of one batch sample per calculation) and random gradient descent (a randomly selected sample to compute the loss).
PS: The SGD (random gradient descent) is now referred to as mini-batch-gradient-descent (batch gradient descent), followed by GN instead of l′ (θn)