Optimizer how to realize the weight of neural network, the updating of migration coefficients and the calculation of gradients in TensorFlow

Case code:

#建立抽象模型
x = Tf.placeholder (Tf.float32, [None, 784])
y = Tf.placeholder (Tf.float32, [None, ten]) #实际分布的概率值


w = tf. Variable (Tf.zeros ([784, 10])
b = tf. Variable (Tf.zeros (10))
A = Tf.nn.softmax (Tf.matmul (x, W) + b) #基于softmax多分类得到的预测概率


#定义损失函数和训练方法
Cross_entropy = Tf.reduce_mean (-tf.reduce_sum (Y * tf.log (a), reduction_indices=[1])) #交叉熵
Optimizer = Tf.train.GradientDescentOptimizer (0.5) #梯度下降优化算法, learning step is 0.5
Train = Optimizer.minimize (cross_entropy) #训练目标: Minimizing loss function


init = Tf.global_variables_initializer ()


Print (' Start to run session: ')


With TF. Session () as Sess:
Sess.run (INIT)
For I in Range (2000):
Batch_xs, Batch_ys = Mnist.train.next_batch (100)
Sess.run (Train, Feed_dict={x:batch_xs, Y:batch_ys})


#test trained model
Correct_prediction = Tf.equal (Tf.argmax (A, 1), Tf.argmax (Y, 1))
accuracy = Tf.reduce_mean (Tf.cast (correct_prediction, Tf.float32))
Print (Sess.run (accuracy, feed_dict={x:mnist.test.images, y:mnist.test.labels}))

The first step:

Neural network model coefficients W, b declares variable variables, where default trainable=true, then these variable variables are automatically placed in the Graphkey.trainable_variables list of the TensorFlow system:

Defaults to the list of variables collected in the graph under the key ' Graphkey.trainable_variables '.

After the continuous optimization of the objective function, these variables are updated by the new gradient during the gradient calculation, and the target of the weight coefficient update is achieved.

Step Two:

Cross_entropy = Tf.reduce_mean (-tf.reduce_sum (Y * tf.log (a), reduction_indices=[1])) #交叉熵
Optimizer = Tf.train.GradientDescentOptimizer (0.5) #梯度下降优化算法, learning step is 0.5
Train = Optimizer.minimize (cross_entropy) #训练目标: Minimizing loss function

Here define the loss function, the optimization algorithm and the final training model of the three operation, the existence of the task dependencies, in TensorFlow graph these operation will form the preservation of dependencies, the final session execution train This operation, will be based on the dependency, search forward, find the earliest operation, start step-by-step execution, the earliest operation that is w B, such as the declaration of these op.

In the Optimizer.minimize function, the main function is to perform two functions:

Compute_gradients functions and Apply_gradients functions

Compute_gradients the variable in the var_list (the variable in Graphkey.trainable_variables is not specified by default), calculates the loss gradient

The apply_gradients effect is to use the computed gradient to update the variables in the var_list, and if no var_list is specified, the variables in the graphkey.trainable_variables are updated