The gradient descent method (gradient descent), aka steepest descent (steepest descent), is the most common method for solving unconstrained optimization problems, it is an iterative method, and the main operation of each step is to solve the gradient vectors of the objective function. The negative gradient direction of the current position is used as the search direction (because the target function falls fastest in that direction, which is also the origin of the steepest descent method name).
Gradient Descent method features: The closer to the target value, the smaller the step, the slower the descent rate.
Visually, as shown in:
Here each circle represents a function gradient, the most central representation of the function extreme point, each iteration based on the current position of the gradient (used to determine the search direction and with the stride speed together to determine the pace) and the step to find a new position, so that the iteration eventually reached the target function local optimal point (if the objective function is a convex function, The global optimal point is reached).
Below we will use the formula to specify the gradient descent method
The following H (θ) is our fitting function
It can also be expressed in vector form:
The following function is the risk function that we need to optimize, each of which represents the residuals between our fitted function and Y on the existing training set, and calculates its square loss function as the risk function we construct (see least squares and their python implementations)
Here we multiply 1/2 is to facilitate the subsequent partial derivative when the result is more concise, the reason can multiply 1/2 is because of this coefficient after the solution to the optimal value of the risk function has no effect.
Our goal is to minimize the risk function so that our fitting function can fit the target function y to the maximum extent:
The following specific gradient solution is carried out around this goal.
Batch gradient descent bgd
According to the traditional idea, we need to obtain the partial derivative of each of the above risk functions, and get each corresponding gradient
This represents the J component of the first sample point, i.e. h (θ).
Next, because we want to minimize the risk function, we update each one in the negative gradient direction of each parameter.
The alpha here indicates the step size
From the above formula can be noted that it is a global optimal solution, but each iteration step, will be used to the training set all the data, if M is large, then imagine this method of iteration speed!! So, this introduces another method, a random gradient descent.
random gradient descent sgd
Because the batch gradient descent is very slow in the case of a large training set, it is not feasible to use the batch gradient descent to solve the optimization problem of the risk function in this case, and in this case, the stochastic gradient descent is proposed.
We rewrite the risk function described above into the following form:
which
A loss function called a sample point
Next we have a loss function for each sample, to each of the partial derivative, to get each corresponding gradient
Each parameter is then updated according to the negative gradient direction of each
Compared to the batch gradient descent, the random gradient descent uses only one sample per iteration, and in the case of large sample sizes, it is common to iterate the θ to the optimal solution using only a subset of the sample data. Therefore, the decrease of random gradient is much less than that of batch gradient reduction in computational amount.
One drawback of SGD is that it is more noisy than BGD, making SGD not each iteration toward the overall optimization direction. and SGD because each time uses a sample to iterate, so the final optimal solution is often not the global optimal solution, but only the local optimal solution. But the direction of the large whole is to the global optimal solution, the final result is often near the global optimal solution.
Here are the graphical presentations of the two methods:
As can be seen from the above figure, SGD because each time a sample point gradient search, so its optimal path looks more blind (this is also the origin of the random gradient descent name).
Compare the advantages and disadvantages of the following points:
Batch gradient drop:
Advantages: Global optimal solution, easy to implement in parallel, not much iteration number
Disadvantages: When the number of samples is large, the training process is slow, and each iteration takes a lot of time.
random gradient descent:
Advantages: Fast training speed with little calculation per iteration
Disadvantages: Reduced accuracy, not global optimal, not easy to implement in parallel, the total number of iterations is more.
============ Split Segmentation =============
Above we explain what is gradient descent and how to solve the gradient descent, we will use Python to achieve gradient descent method.
Results (interception section):
You can see the final convergence to a stable parameter value.
Note: This is a careful choice when selecting Alpha and epsilon, which may cause the final failure to converge.
Reference Documentation:
A gradient descent algorithm for Python
(turn) Gradient descent method and its Python implementation
Start building with 50+ products and up to 12 months usage for Elastic Compute Service
1 on 1 presale consultation
24/7 Technical Support 6 Free Tickets per Quarter Faster Response
Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.
A comprehensive suite of global cloud computing services to power your business
Payment Methods We Support
