Knowledge preparation for gradient descent in deep learning

Consider a cost function, C, which calculates the cost of the current iteration model based on the parameter vectors and is written as C (). In machine learning, our task is to get the minimum value of the cost, which is usually the mean value of the loss function or its mathematical expectation in machine learning. See:

This is called the generalization loss, and in the process of supervision we know z= (x, y) and f (×) is the prediction of Y.

What is the gradient here?

when it is scalar, the gradient of the cost function can be expressed as follows:

When It's very small, it's another expression, and we're making it less than 0, and the smaller the better.

At that time a vector, the gradient of the cost function is also a vector, each is a I, here I consider the other variables are fixed, is only in the change. When it is small, it becomes

Then look at the gradient drop:

Here Our goal is to find a  . If we can handle this, we'll have a big fork,

So we're going to find the best model. However, the ideal is very plump, the reality is very bony, in 99.99999% of the situation, we can not find the answer to the above equation, unless the computer is human, so we use mathematical optimization method. Most methods are the idea of local descent. Each round of iteration adjusted under    , to reduce the value of the cost function until the cost function cannot be lowered, so that we reach a local minimum (if we are lucky, I might find the global minimum).

The simplest gradient-based optimization technique is gradient descent. There are many methods of gradient descent, and we are talking about the most common method here.

Here represents the iteration of the K-theory,     is the learning rate, it is a scalar. You can use a fixed learning rate, or you can use the appropriate learning rate, which is not detailed here.

Let's look at a random gradient descent method ( stochastic Gradient descent

We know that C is a mean, general and specific sample independent, if we update a bit faster, in extreme cases, each sample updates, we have the following formula:

, where z is the next sample, or the next input signal for online learning. SGD is a more general principle, and its gradient descent direction is relatively random, in other words, is declining in the direction of interest. SGD is basically similar to a normal gradient drop, except that it adds a lot of randomness.

SGD drops faster than a normal gradient (batch gradient descent) because it updates the weight vectors more frequently. This is useful for large datasets, or for online learning. In fact, in the machine learning task, we only use the batch gradient descent method when the cost function cannot be decomposed into the above equation.

Low-Volume gradient descent method:

This does not want to be careful, that is, each training 10 or 20 drops once, the method and the batch of the same.

Moving averages Method:

Instead of using the current gradient of the sample, we calculate the moving average of a past sample and then use the moving average to do the weight vector adjustment.

-------------------------------the last gorgeous split line------------------------------------------

For this tutorial, please click here for the catalogue blog.

If you want to get started with machine learning first, here 's a quick introduction. If you need a quick overview of deep learning, here 's a simple introduction. Before learning these tutorials, you can warm up first, here is the basic tutorial of Theano, after learning, then look at this stuff, there are some basic concepts and some test training set.

Knowledge preparation for gradient descent in deep learning