Gradient cropping of TensorFlow

In deeper networks, such as multilayer CNN or very long rnn, the problem of gradient disappearance (Gradient vanishing) or gradient explosion (Gradient exploding) may occur due to the chain rule of derivation.

Principle

question : Why does a gradient explosion cause instability in training and does not converge?
Gradient explosion, in fact, is a large derivative of the meaning. Recall that we used the gradient descent method to update the parameters:

The value of the loss function decreases along the direction of the gradient, however, if the gradient (partial derivative) is very big, the function value jumps to jump, the convergence is not the most value,

Of course this happens, one of the solutions is to set the learning rate αα to a smaller point, such as 0.0001.

This paper introduces the method of gradient cropping (Gradient Clipping), cuts the gradient, and proposes to cut the L2 norm of the gradient, that is, the square and re-prescribing of the partial derivative of all parameters.

TensorFlow Code

Method One:

Optimizer = Tf.train.AdamOptimizer (learning_rate=0.001, beta1=0.5= optimizer.compute_gradients (loss)   for in enumerate (grads):    if is not None:        = (Tf.clip_by_norm (g, 5), V)  #  threshold is set here to 5train_op = optimizer.apply_gradients (grads)

which
optimizer.compute_gradients()Returns the normal computed gradient, which is a list containing (gradient, variable).

tf.clip_by_norm(t, clip_norm)Returns the cropped gradient, which is the same dimension as T.

It is important to note, however, that the norm here is not based on the global gradient, but on the part.

Method Two:

Optimizer = Tf.train.AdamOptimizer (learning_rate=0.001, beta1=0.5= Zip (*= Tf.clip_by_global_norm ( Grads, 5= optimizer.apply_gradients (Zip (grads, variables))

Here is the calculation of the global norm, which is standard. The downside, though, is that it can be slowed down, because all gradients need to be calculated before cropping can be done.

Summarize

Gradient cropping is also a good method when you train a model to have a loss value that does not converge, except to set a small learning rate.

However, it also shows that if your model is stable and convergent, but ineffective, then it has nothing to do with the learning rate and the gradient explosion. Therefore, the setting of learning rate and the threshold of gradient clipping can not improve the accuracy of the model.

Gradient cropping of TensorFlow