"TensorFlow" Queue & multithreaded &tfrecod file _ my generation when singing

A detailed description of the TF data read queue mechanism

• TFR file multi-threaded queue read and write operations:

• • Tfrecod File Write operation:

Import TensorFlow as Tfdef _int64_feature (value):    # value must be an iterator object    # non-int data use bytes instead of Int64 to    return Tf.train.Feature (Int64_list=tf.train.int64list (Value=[value])) Num_shards = 2instance_perpshard = 2for i in range (num_ Shards):    filename = (' ftr/data.tfrecords-%.5d-of-%.5d '% (i, num_shards))    writer = Tf.python_io. Tfrecordwriter (filename)                      #<---------writer Open for    J in Range (Instance_perpshard):        example = Tf.train.Example (Features=tf.train.features (feature={#<---------write into buffer            ' I ': _int64_feature (i),            ' J ': _ Int64_feature (j)        })        Writer.write (example. Serializetostring ())                       #<---------written into the actual file    writer.close ()                                                      #<---------writer off

• • Tfrecod file read operation:

Default multi-threading, this default multithreaded procedure for maintaining file name queues

' Read TFR ' files = [' ftr/data.tfrecords-00000-of-00002 ', ' ftr/data.tfrecords-00001-of-00002 ']# files = Tf.train.match_filenames_once ("ftr/data.tfrecords-*") # Input file Name List # return Queuerunner & fifoqueue# scrambled order & join queue and Output queue get file belongs to separate thread filename_queue = tf.train.string_input_producer (Files, shuffle=false) #<---------input file Queue reader = TF . Tfrecordreader () #<---------reader Open _,serialized_example = Reader.read (filename_queue) #<------        ---read the original file features = Tf.parse_single_example (#<---------read parsed file serialized_example, features={ ' I ': TF. Fixedlenfeature ([],tf.int64), ' J ': TF. Fixedlenfeature ([],tf.int64)}) with TF. Session () as Sess:tf.global_variables_initializer (). Run () coord = Tf.train.Coordinator () #<---------multithreading Threa ds = Tf.train.start_queue_runners (sess=sess,coord=coord) #<---------file name queue populate thread start for I in range (6): Print (Sess . Run ([features[' I '],features[' J '])) #< starts the read process coord---------the actual session.Request_stop () #<---------multithreading coord.join (Threads) #<---------Multithreading 

• • Tfrecod File Packaging operations:

Packaging mechanism:

—————— multithreading calls the previous node to calculate the queue

—————— Batch out and pack

So do not need to modify parsing read data process for loops and so on, can be said to be very convenient

Example_batch, Label_batch = Tf.train.batch ([example, label],     #<---------multithreaded batch generation                                            Batch_size=batch_ Size,                                            num_threads=3,                                            capacity=capacity)

example_batch, Label_batch = Tf.train.shuffle_batch ([example, label], #<-                                            --------multithreaded Random batch generation Batch_size=batch_size,                                            Num_threads=3, Capacity=capacity, 
 MIN_AFTER_DEQUEUE=30) because the element is too few random meaning is not big, so many more parameters 

Files = ["ftr/data.tfrecords-00000-of-00002", "ftr/data.tfrecords-00001-of-00002"]# files = tf.train.match_filenames _once ("ftr/data.tfrecords-*") # Input file Name List # return Queuerunner & fifoqueue# scrambled order & join queue and output queue get file belongs to separate thread filename_queue = Tf.train.string_input_producer (Files, shuffle=false) #<---------input file Queue reader = tf. Tfrecordreader () #<---------Read _,serialized_example = Reader.read (filename_queue) #<--------- Read features = Tf.parse_single_example (#<---------read serialized_example, features={' i ': TF. Fixedlenfeature ([],tf.int64), ' J ': TF. Fixedlenfeature ([],tf.int64)}) example, label = Features[' i '], features[' j ']batch_size = 2capacity = + 3 * batch_s ize# a single sample, out of the queue batch# can specify multiple threads to perform a queued operation simultaneously example_batch, Label_batch = Tf.train.batch ([example, label], #<---------multithreading b Atch generate Batch_size=batch_size, num_th                   Reads=3,                         capacity=capacity) with TF.                                Session () as Sess:tf.global_variables_initializer (). Run () coord = Tf.train.Coordinator () #<---------multithreaded Manager threads = tf.train.start_queue_runners (sess=sess,coord=coord) #<---------file name queue populate thread start fo R I in range (3): Cur_example_batch, Cur_label_batch = Sess.run ([Example_batch, Label_batch]) print (Cur_examp Le_batch, Cur_label_batch) coord.request_stop () #<---------multithreading off COORD.J                          Oin (Threads)

This output is an image (a surrogate) before each line, followed by a label, the first row of data pair is actually 0-0, 0-1:

[0 0] [0 1] [1 1] [0 1] [0 0] [0 1]

• Picture files use TFR read-write test:

Read binary data directly _bytes_feature can be written to the file, using the tf.string type to read the image data can be directly decode decoding (presumably TF string corresponding to binary data type).

Write a picture into the TFR:

Import TensorFlow as Tfimport Matplotlib.pyplot as Pltdef _bytes_feature (value):    return Tf.train.Feature (bytes_ List=tf.train.byteslist (Value=[value]) def _int64_feature (value):    return Tf.train.Feature (int64_list= Tf.train.Int64List (Value=[value])) Img_raw = Tf.gfile.FastGFile (' 123123.jpeg ', ' RB '). Read () filename = (' ftr/ Image.tfrecords ') writer = Tf.python_io. Tfrecordwriter (filename) #<---------Writing Example = Tf.train.Example (features=tf.train.features (feature={#<---- -----Write    ' image ': _bytes_feature (Img_raw),    ' label ': _int64_feature (1)    })) Writer.write (example. Serializetostring ()) #<---------writing writer.close ()

Read the picture data from the TFR and decode the drawing out:

filename_queue = Tf.train.string_input_producer ([' Ftr/image.tfrecords '], Shuffle=false) #<---------input file Queue reader = tf. Tfrecordreader () #<---------Read _,serialized_example = Reader.read (filename_queue) #<--------- Read features = Tf.parse_single_example (#<---------read serialized_example, features={' image ' : TF. Fixedlenfeature ([],tf.string), ' label ': TF. Fixedlenfeature ([],tf.int64)}) img = tf.image.decode_jpeg (features[' image ')) with TF. Session () as Sess:tf.global_variables_initializer (). Run () coord = Tf.train.Coordinator () # <---------multithreading THR EADS = Tf.train.start_queue_runners (sess=sess, Coord=coord) # <---------file name queue populate thread start # img_raw, label = Sess.run ([fe  atures[' image '], features[' label ']) image = Sess.run (img) plt.imshow (image) Plt.show () Coord.request_stop () # <---------multithreading Coord.join (Threads) # <---------multithreading 

• Picture files use queue Read and write operations directly:

Only demonstrates the maintenance of the picture file name queue Read and write, there are not too many other operations

reader = TF. Wholefilereader (): New reader, should be norm binary file reader

# import Tensorflowimport TensorFlow as tf# create a new Sessionwith TF. Session () as Sess:    # We're going to read three pictures a.jpg, b.jpg, c.jpg    filename = [' 123.png ', ' 123123.jpeg ']    # String_input_ Producer will produce a file name queue    filename_queue = tf.train.string_input_producer (filename, shuffle=false, num_epochs=5)    # Reader reads data from the file name queue. The corresponding method is reader.read    reader = tf. Wholefilereader ()                             #<---------Note that the reader is different.    Key, value = Reader.read (filename_queue)    # Tf.train.string_ Input_producer defines an epoch variable to initialize it with    Tf.local_variables_initializer (). Run ()    # after using Start_queue_runners, To start populating the queue    threads = tf.train.start_queue_runners (sess=sess)    i = 0 while    True:        i + = 1        # Get picture data and save        Image_data = Sess.run (value) with        open (' test_%d.jpg '% i, ' WB ') as F:            F.write (Image_data)

• Example of a queue file used in a book:

File name queue creation, read parse file, package parsing files, multi-threaded start-up graph training (multithreading refers to the part that is used is actually a file read)

Import TensorFlow as tf ' Create file list ' ' Files = tf.train.match_filenames_once ("records/output.tfrecords") Filename_queue = Tf.train.string_input_producer (Files, Shuffle=false) "Parses the data in the Tfrecord file" # to read the file. reader = TF. Tfrecordreader () _,serialized_example = Reader.read (filename_queue) # parse the Read sample. features = Tf.parse_single_example (Serialized_example, features={' Image_raw ': TF. Fixedlenfeature ([],tf.string), ' pixels ': tf. Fixedlenfeature ([],tf.int64), ' label ': TF. Fixedlenfeature ([],tf.int64)}) Decoded_images = Tf.decode_raw (features[' Image_raw '],tf.uint8) retyped_images = Tf.cast (decoded_images, tf.float32) labels = tf.cast (features[' label '],tf.int32) #pixels = tf.cast (features[' pixels ') , tf.int32) images = Tf.reshape (Retyped_images, [784]) "pack the files in 100 for a group" "Min_after_dequeue = 10000batch_size =                                                    100capacity = min_after_dequeue + 3 * batch_sizeimage_batch, Label_batch = Tf.train.shuffle_batch ([images, labels], Batch_size=batch_Size, capacity=capacity, Min_after_dequeue=min_after_dequeue) ' Training model ' def inference (Input_tensor, Weights1, Biases1, Weights2, biases2): l Ayer1 = Tf.nn.relu (Tf.matmul (Input_tensor, weights1) + biases1) return Tf.matmul (Layer1, weights2) + biases2# model-related parameters in Put_node = 784output_node = 10layer1_node = 500regularaztion_rate = 0.0001training_steps = 5000weights1 = tf. Variable (Tf.truncated_normal ([Input_node, Layer1_node], stddev=0.1)) Biases1 = tf. Variable (Tf.constant (0.1, Shape=[layer1_node])) weights2 = tf. Variable (Tf.truncated_normal ([Layer1_node, Output_node], stddev=0.1)) Biases2 = tf. Variable (Tf.constant (0.1, Shape=[output_node])) y = Inference (Image_batch, Weights1, Biases1, Weights2, Biases2) # Calculate cross entropy and its average cross_entropy = Tf.nn.sparse_softmax_cross_entropy_with_logits (logits=y, Labels=label_batch) Cross_ Entropy_mean = Tf.reduce_mean (cross_entropy) # loss function Calculation Regularizer = tf.contrib.laYers.l2_regularizer (regularaztion_rate) regularaztion = Regularizer (weights1) + regularizer (weights2) loss = Cross_ Entropy_mean + regularaztion# optimization loss function Train_step = Tf.train.GradientDescentOptimizer (0.01). Minimize (loss) # Initialize the session and start the training process. With TF. Session () as Sess:tf.global_variables_initializer (). Run () coord = Tf.train.Coordinator () threads = Tf.train.star    T_queue_runners (Sess=sess, Coord=coord) # Loops the training neural network. For I in Range (training_steps): if I% = = 0:print ("After%d TRAINING step (s), loss is%g"% (I, S Ess.run (loss)) Sess.run (Train_step) coord.request_stop () Coord.join (threads)

"TensorFlow" Queue & multithreaded &tfrecod file _ my generation when singing