Gradient Descent Algorithm

Early Exposure to Gradient DescentAlgorithmBut I seem to understand it. This is indeed related to the gradient, but I think the relationship is not that big. If I understand the significance of the derivative, I can fully understand the algorithm. There is good information on the Internet. I will reference it here. Give a picture first

We know that the current minimum point is at point B,

If I know that the partial derivative of Point A is less than 0, that is, the slope of Point A is less than 0, I should find B on the right of, that is, the opposite number of A's X coordinate + A's derivative.

If I know that the partial derivative of point B is greater than 0 at point B, then B should be on the left of point C, that is, the opposite number of the derivative of point B + C.

Now we have understood the basic idea. In fact, it is to give a current vertex and find the partial derivative of the current vertex, so that the value of the next independent variable is: the value of the current independent variable + the opposite number of partial derivatives of the current vertex. To improve efficiency, we generally multiply the value of the current independent variable + the opposite number of partial derivatives of the current vertex by a step. When the function value between two consecutive points is less than a certain threshold value, the minimum value is basically determined.

This is a 2-dimensional image that is easy to understand, as is the case for multi-dimensional images. Here is a MATLAB demonstration of x ^ 2 + y ^ 2.Code, You can directly see how the vertex is doing iteration close to the minimum value.

The specific Matlab code is as follows:

% The image of the most primitive equation is drawn. The equation is f = x ^ 2 + y ^ 2. the preset domain is set to [-10, 10], respectively. set an initial vertex and iteration value and step to view the iteration process using the gradient descent algorithm. % [x, y] = meshgrid.. 5: 10); Z = x. ^ 2 + Y. ^ 2; mesh (x, y, z); % returns the partial derivative of X and Y. % dx = 2 * X; % DY = 2 * Y; % sets X, y, and the updated values in each step are added to this record. The first value is the initial value. For iteration convenience, set the record_values = [-9; -9]; % set the step % step = 0.2 for each update; % set the number of iterations % COUNT = 20; % start iteration % for I = 1: Count current_x = record_values (1, i); current_y = record_values (2, I); temp = [current_x-step * 2 * current_x, current_y-step * 2 * current_y]; % append temp to % record_values after record_values (:, I + 1) = temp; end % to calculate the points and draw them out. These points are connected in order. % Hold on; x_values = record_values (1, :); y_values = record_values (2, :); z_values = x_values. * x_values + y_values. * y_values; plot3 (x_values, y_values, z_values, 'K -- o'); % We know that when X and Y are equal to 0, it is the minimum vertex and % plot3 (0, 0, 0, 'rp ');
 

Disadvantage: in fact, according to the above analysis, we can know that in order to achieve better results, the step size and convergence value cannot be too large, especially the step size. Otherwise, we will select between two points, so that the optimal solution cannot be found. For non-convex problems, we may find that only the local minimum value is obtained.