The method of TensorFlow to realize random training and batch training

This article mainly introduced the TensorFlow realizes the random training and the batch training method, now shares to everybody, also gives everybody to make a reference. Come and see it together.

TensorFlow Update model variables. It can manipulate one data point at a time, or it can manipulate large amounts of data at once. An operation on a training example may lead to a more "eccentric" learning process, but using a large amount of training can result in expensive computational costs. The choice of training type is critical to the convergence of machine learning algorithms.

In order to tensorflow the variable gradient to allow the reverse propagation to work, we must measure the loss of one or more samples.

Random training will be a random sampling of training data and target data to complete the training. Another option is to have the average loss for a large batch of training to perform a gradient calculation, and the bulk training size can be extended to the entire data set at once. Here's an example of how to extend the previous regression algorithm-using random training and batch training.

The difference between batch training and random training is their optimizer approach and convergence.

# Random Training and Batch training #----------------------------------# # This Python function illustrates the different training methods:# batch and stochastic training. For each model, we'll use# a regression model that predicts one model Variable.import Matplotlib.pyplot as Pltimport num Py as Npimport TensorFlow as Tffrom tensorflow.python.framework import opsops.reset_default_graph () # Random training: # Create graphs ESS = TF. Session () # declaration Data X_vals = np.random.normal (1, 0.1, +) Y_vals = Np.repeat (Ten) X_data = Tf.placeholder (shape=[1], Dtype =tf.float32) Y_target = Tf.placeholder (shape=[1], Dtype=tf.float32) # declares A variable (one model parameter = a) a = tf. Variable (Tf.random_normal (shape=[1])) # Add operation to Figure My_output = tf.multiply (X_data, A) # Add L2 loss Function loss = Tf.square (my_output-y _target) # Initialize variable init = Tf.global_variables_initializer () sess.run (INIT) # declaration Optimizer my_opt = Tf.train.GradientDescentOptimizer (0.02) Train_step = my_opt.minimize (loss) loss_stochastic = []# run iteration for I in range (100) : Rand_index = Np.random.choice (+) rand_x = [x_vals[Rand_index]] rand_y = [Y_vals[rand_index]] Sess.run (Train_step, feed_dict={x_data:rand_x, y_target:rand_y}) if (i+1)%5 ==0:print (' Step # ' + str (i+1) + ' a = ' + str (Sess.run (a))) Temp_loss = Sess.run (loss, feed_dict={x_data:rand_x, Y_tar Get:rand_y}) Print (' Loss = ' + str (temp_loss)) Loss_stochastic.append (temp_loss) # Batch training: # Reset calculation diagram Ops.reset_default_graph ( ) Sess = tf. Session () # Declaring bulk size # Batch size refers to how much training data is passed through the calculation chart at a time batch_size = 20# Declaration model data, Placeholder x_vals = np.random.normal (1, 0.1, +) Y_vals = Np.rep Eat (X_data) = Tf.placeholder (Shape=[none, 1], dtype=tf.float32) Y_target = Tf.placeholder (Shape=[none, 1], Dtype =tf.float32) # declares A variable (one model parameter = a) a = tf. Variable (Tf.random_normal (shape=[1,1)) # adds matrix multiplication operation (matrix multiplication does not meet commutative law) My_output = Tf.matmul (X_data, A) # Add loss function # The loss function in bulk training is the average of L2 loss per data point loss = Tf.reduce_mean (Tf.square (my_output-y_target)) # Initialize variable init = Tf.global_variables_ Initializer () Sess.run (init) # claims Optimizer my_opt = Tf.train.GradientDescentOptimizer (0.02) Train_step = my_opt.minimize (loss ) Loss_Batch = []# run iteration for I in range: Rand_index = Np.random.choice (size=batch_size) rand_x = Np.transpose ([X_vals[ran D_index]] rand_y = Np.transpose ([Y_vals[rand_index]]) sess.run (Train_step, feed_dict={x_data:rand_x, Y_target:rand_  Y}) if (i+1)%5==0:print (' Step # ' + str (i+1) + ' a = ' + str (Sess.run (a))) Temp_loss = Sess.run (loss, Feed_dict={x_data: rand_x, y_target:rand_y}) print (' Loss = ' + str (temp_loss)) loss_batch.append (Temp_loss) plt.plot (range (0, 5), Los S_stochastic, ' B ', label= ' stochastic Loss ') Plt.plot (range (0, 5), Loss_batch, ' r--', label= ' batch Loss, size=20 ') Plt.legend (loc= ' upper right ', prop={' size ': one}) Plt.show ()

Output:

Step #5 A = [1.47604525]
Loss = [72.55678558]
Step #10 A = [3.01128507]
Loss = [48.22986221]
Step #15 A = [4.27042341]
Loss = [28.97912598]
Step #20 A = [5.2984333]
Loss = [16.44779968]
Step #25 A = [6.17473984]
Loss = [16.373312]
Step #30 A = [6.89866304]
Loss = [11.71054649]
Step #35 A = [7.39849901]
Loss = [6.42773056]
Step #40 A = [7.84618378]
Loss = [5.92940331]
Step #45 A = [8.15709782]
Loss = [0.2142024]
Step #50 A = [8.54818344]
Loss = [7.11651039]
Step #55 A = [8.82354641]
Loss = [1.47823763]
Step #60 A = [9.07896614]
Loss = [3.08244276]
Step #65 A = [9.24868107]
Loss = [0.01143846]
Step #70 A = [9.36772251]
Loss = [2.10078788]
Step #75 A = [9.49171734]
Loss = [3.90913701]
Step #80 A = [9.6622715]
Loss = [4.80727625]
Step #85 A = [9.73786926]
Loss = [0.39915398]
Step #90 A = [9.81853104]
Loss = [0.14876099]
Step #95 A = [9.90371323]
Loss = [0.01657014]
Step #100 A = [9.86669159]
Loss = [0.444787]
Step #5 A = [[2.34371352]]
Loss = 58.766
Step #10 A = [[3.74766445]]
Loss = 38.4875
Step #15 A = [[4.88928795]]
Loss = 27.5632
Step #20 A = [[5.82038736]]
Loss = 17.9523
Step #25 A = [[6.58999157]]
Loss = 13.3245
Step #30 A = [[7.20851326]]
Loss = 8.68099
Step #35 A = [[7.71694899]]
Loss = 4.60659
Step #40 A = [[8.1296711]]
Loss = 4.70107
Step #45 A = [[8.47107315]]
Loss = 3.28318
Step #50 A = [[8.74283409]]
Loss = 1.99057
Step #55 A = [[8.98811722]]
Loss = 2.66906
Step #60 A = [[9.18062305]]
Loss = 3.26207
Step #65 A = [[9.31655025]]
Loss = 2.55459
Step #70 A = [[9.43130589]]
Loss = 1.95839
Step #75 A = [[9.55670166]]
Loss = 1.46504
Step #80 A = [[9.6354847]]
Loss = 1.49021
Step #85 A = [[9.73470974]]
Loss = 1.53289
Step #90 A = [[9.77956581]]
Loss = 1.52173
Step #95 A = [[9.83666706]]
Loss = 0.819207
Step #100 A = [[9.85569191]]
Loss = 1.2197

Training Type	Advantages	Disadvantages
Random Training	Out of the local minimum	It usually takes more iterations to converge.
Batch Training	Get the minimum loss quickly	Consume more computing resources