The specific use of next_batch in tensorflow,

The specific use of next_batch in tensorflow,

This article introduces the use of next_batch in tensorflow and shares it with you as follows:

Several different next_batch methods are provided here. This article only explains the code snippets for future reference:

Def next_batch (self, batch_size, fake_data = False): "" Return the next 'batch _ size' examples from this data set. "if fake_data: fake_image = [1] * 784 if self. one_hot: fake_label = [1] + [0] * 9 else: fake_label = 0 return [fake_image for _ in xrange (batch_size)], [fake_label for _ in xrange (batch_size)] start = self. _ index_in_epoch self. _ index_in_epoch + = batch_size if self. _ index_in_epoch> self. _ num_examples: # indicates whether the sentence subscript in epoch is greater than the number of all corpus. If it is True, a new round of traversal begins # Finished epoch self. _ epochs_completed + = 1 # Shuffle the data perm = numpy. arange (self. _ num_examples) # The arange function is used to create an equality array numpy. random. shuffle (perm) # disrupt self. _ images = self. _ images [perm] self. _ labels = self. _ labels [perm] # Start next epoch start = 0 self. _ index_in_epoch = batch_size assert batch_size <= self. _ num_examples end = self. _ index_in_epoch return self. _ images [start: end], self. _ labels [start: end]

This section of code is taken from mnist. py file, starting from the code 12th line = self. _ index_in_epoch start to explain, _ index_in_epoch-1 is the bottom of the last batch image, the subscript of the first epoch image is from _ index_in_epoch, the subscript of the last image is _ index_in_epoch + batch. If _ index_in_epoch is greater than the number of images in the corpus, it indicates that this epoch is inappropriate, even if it completes the traversal of the corpus again, so we should shuffles the image and start a new round of corpus composition.

Def ptb_iterator (raw_data, batch_size, num_steps): "" Iterate on the raw PTB data. this generates batch_size pointers into the raw PTB data, and allows minibatch iteration along these pointers. args: raw_data: one of the raw data outputs from ptb_raw_data. batch_size: int, the batch size. num_steps: int, the number of unrolls. yields: Pairs of the batched data, each a matrix of shape [batch_size, num_steps]. the second element of the tuple is the same data time-shifted to the right by one. raises: ValueError: if batch_size or num_steps are too high. "raw_data = np. array (raw_data, dtype = np. int32) data_len = len (raw_data) batch_len = data_len // batch_size # How many batch data = np. zeros ([batch_size, batch_len], dtype = np. int32) # How many words does batch_len have for I in range (batch_size): # How many batch_size batchdata [I] = raw_data [batch_len * I: batch_len * (I + 1)] epoch_size = (batch_len-1) // num_steps # batch_len indicates the number of sentences in a batch # epoch_size = (len (data) // model. batch_size)-1) // model. num_steps # // indicates the integer division if epoch_size = 0: raise ValueError ("epoch_size = 0, decrease batch_size or num_steps") for I in range (epoch_size ): x = data [:, I * num_steps :( I + 1) * num_steps] y = data [:, I * num_steps + 1 :( I + 1) * num_steps + 1] yield (x, y)

Method 3:

  def next(self, batch_size):    """ Return a batch of data. When dataset end is reached, start over.    """    if self.batch_id == len(self.data):      self.batch_id = 0    batch_data = (self.data[self.batch_id:min(self.batch_id +                         batch_size, len(self.data))])    batch_labels = (self.labels[self.batch_id:min(self.batch_id +                         batch_size, len(self.data))])    batch_seqlen = (self.seqlen[self.batch_id:min(self.batch_id +                         batch_size, len(self.data))])    self.batch_id = min(self.batch_id + batch_size, len(self.data))    return batch_data, batch_labels, batch_seqlen

Method 4:

Def batch_iter (sourceData, batch_size, num_epochs, shuffle = True): data = np. array (sourceData) # convert sourceData to array Storage data_size = len (sourceData) num_batches_per_epoch = int (len (sourceData)/batch_size) + 1 for epoch in range (num_epochs ): # Shuffle the data at each epoch if shuffle: shuffle_indices = np. random. permutation (np. arange (data_size) shuffled_data = sourceData [partition] else: shuffled_data = sourceData for batch_num in range (partition): start_index = batch_num * batch_size end_index = min (batch_num + 1) * batch_size, data_size) yield shuffled_data [start_index: end_index]

The usage of the iterator. Learn more about the usage of the Python iterator.

Note that the first three methods only traverse all the corpus. The last method is that all the corpus traverses num_epochs.

The above is all the content of this article. I hope it will be helpful for your learning and support for helping customers.