Analysis of Gradient descent method

The gradient descent method is involved in the Solver optimization method of learning Caffe,caffe in the last period. At that time, the concept of gradient descent method and the principles are very vague, specifically to learn the next, now put their own understanding of the record, on the one hand to deepen the impression, on the one hand is also convenient to consult at any time. If there is an understanding of the wrong place, want to see to correct, thank you.

　　first, what is the gradient? What is the relationship between gradients and the number of square wizards? (briefly, you can search by yourself for more information)

Party wizards: For a function F, there is a bit of Kin its definition field, we put the derivative of function F in the direction of Point K , called the number of square wizards.

Gradient: The mathematical reasoning can prove that the function F in the direction of the K -point gradient, equal to the function f in the K-Point wizard number to take the maximum direction, the gradient modulus is the direction derivative maximum value. Here we can understand that: the number of square wizards represents the function at the K-point conversion rate, when the maximum transformation rate, the longest gradient, the function at the point of the gradient direction is upward.

　　Second, loss function--loss functions

The loss function describes the extent to which the predictive function is "bad".

Suppose for a set of Datasets (X1,y1), (x2,y2),..., (Xn,yn), assuming that its predictive function is hθ (x), θ is a parameter weight, is a vector, X is a parameter, and its loss function can be defined as:

where hθ (xi) corresponds to the predicted value of Xi, and Yi is its corresponding true value.

Our aim: To find the one that makes the loss function minimal.

　　Third, how to make the loss function minimum-gradient descent method

　　1, gradient descent method of the process

(1) First, the random assignment;

(2) The derivative is made, and the direction of gradient descent is reduced.

(3) When the gradient falls to the preset value, stop falling. At this point the loss function is considered to be convergent and the rate of change is minimal.

2, Gradient descent method formula:

Meaning: For each, the function changes in the direction in which it drops most, until a minimum is reached.

Understanding: Because the direction of the gradient is the direction of the maximum number of square wizards, the function at the point of the change rate is the largest and positive value, that is, the direction of the current gradient is the most upward direction. If you want to get the most downward descending direction, take the opposite direction of the current gradient direction.

3, example (Note: This example image from http://blog.csdn.net/zengdong_1991/article/details/45563107)

See:

Analysis:

Ps: Personal understanding-because the direction of the gradient is the direction of the maximum number of Square wizard, the maximum number of Square wizard, must be greater than 0, so this time the direction of the gradient is the function of the highest rate of change direction and direction upward. The gradient drop sends a negative sign to change the direction of the gradient to the most downward direction.

　　Iv. gradient descent and gradient rise

1. Gradient Rise formula:

　　

2. Difference and relation between gradient descent and gradient rise

In fact, gradient descent and gradient rise, described in the iterative process of parameter values updated direction . The gradient descent is that the direction of the parameter value update is the opposite of the gradient itself, that is, the downward direction is updated, and the gradient rises that the parameter values are updated in the same direction as the gradient, that is, the upward direction is updated. However, note: regardless of the direction of the gradient is upward or downward, in the iterative process, the gradient modulus is always declining , because with the change in the gradient direction, the change rate of the function is gradually reduced, the change rate is reduced, the maximum value of the number of side wizard decreases, that is, the gradient modulus decreases.

It is not clear, you can refer to the understanding.

PS: After the completion of the discovery of their own understanding is still very simple, in the blog many of the expression is not accurate, first written so, and later understanding deepened, and then revised. Just started to write blog, make very ugly, still need to improve.

Reference blog:

1.http://blog.csdn.net/wolenski/article/details/8030654

2.http://blog.csdn.net/zengdong_1991/article/details/45563107

3.http://blog.csdn.net/xiazdong/article/details/7950084

4.http://www.cnblogs.com/hitwhhw09/p/4715030.html

　　

Analysis of Gradient descent method