TF Boys (TensorFlow Boys) nurturance (II.)

TensorFlow's how-tos, explains these points:

1. Variables: Create, initialize, save, load, share;

2. TensorFlow Visual Learning, (after r0.12 version, added embedding visualization)

3. reading of data;

4. Threads and queues;

5. Distributed TensorFlow;

6. Add a new Ops;

7. Custom data reading;

For a variety of reasons, I only see the first 5 parts, the remaining 2 parts have not time to see, the task is heavy, so hurried off, if useful to the place, then back to study. The learning process is deeply involved in the official documentation of a lot of redundancy, especially the third part of the data read, and smelly and long, it took me a long time, so I would like to organize the third part of the following, convenient for passengers.

TensorFlow There are three ways to read data: 1) to supply data, to read from a file using Placeholder;2), 3) to preload data with constants or variables, for situations where the data size is smaller. Supply data There is nothing to say, have seen before, it is not difficult to understand, we simply said to read from the file data.

Official documents, read data from a file is a long description, links in endless, after reading this link has not read a few words, the next link appears.

It took me a long time to get to know the road, so I want to summarize this part and bring my passengers.

First, you need to know the format of the file you want to read, select the corresponding file reader;

Then, navigate to the Data folder and use the

["file0" "file1"]        #  [("file%d" for in range (2)])    #  tf.train.match_filenames_once

Select the name of the file to read, use the Tf.train.string_input_producer function to generate the file name queue, this function can set shuffle = Ture, to disrupt the queue, you can set the Epoch = 5, over 5 times the training data.

Finally, select the file reader, read the file name queue and decode, enter the Tf.train.shuffle_batch function, generate the batch queue, and pass it to the next layer.

1) If the file you want to read is a text file like CSV, the file reader and decoder are textlinereader and decode_csv.

2) If the data you want to read is a binary file of the. bin format like Cifar10, use TF. Fixedlengthrecordreader and Tf.decode_raw read fixed-length file readers and decoders. My reference code is listed below:

classCifar10_data (object):def __init__(self, filename_queue): Self.height= 32Self.width= 32self.depth= 3self.label_bytes= 1self.image_bytes= Self.height * Self.width *self.depth self.record_bytes= Self.label_bytes +self.image_bytes Self.label, Self.image=self.read_cifar10 (filename_queue)defRead_cifar10 (Self, filename_queue): Reader= TF. Fixedlengthrecordreader (record_bytes =self.record_bytes) key, value=Reader.read (filename_queue) record_bytes=Tf.decode_raw (value, tf.uint8) label=Tf.cast (Tf.slice (Record_bytes, [0], [self.label_bytes]), Tf.int32) Image_raw=Tf.slice (Record_bytes, [self.label_bytes], [self.image_bytes]) Image_raw=Tf.reshape (Image_raw, [Self.depth, Self.height, self.width]) image= Tf.transpose (Image_raw,, 0)) Image =tf.cast (image, Tf.float32)returnlabel, ImagedefInputs (Data_dir, batch_size, train = True, name ='input'): With Tf.name_scope (name):ifTrain:filenames= [Os.path.join (Data_dir,'Data_batch_%d.bin'%II) forIiinchRange (1,6)]             forFinchFilenames:if  nottf.gfile.Exists (f):RaiseValueError ('Failed to find file:'+f) Filename_queue=Tf.train.string_input_producer (filenames) Read_input=Cifar10_data (filename_queue) Images=Read_input.imageImages =tf.image.per_image_whitening (IMAGES)Labels =Read_input.label num_preprocess_threads= 16image, Label=Tf.train.shuffle_batch ([Images,labels], batch_size=batch_size, Num_threads=num_preprocess_threads, Min_after_dequeue= 20000, capacity = 20192)                                returnimage, Tf.reshape (label, [Batch_size])Else: Filenames= [Os.path.join (Data_dir,'Test_batch.bin')]             forFinchFilenames:if  nottf.gfile.Exists (f):RaiseValueError ('Failed to find file:'+f) Filename_queue=Tf.train.string_input_producer (filenames) Read_input=Cifar10_data (filename_queue) Images=Read_input.imageImages =tf.image.per_image_whitening (IMAGES)Labels =Read_input.label num_preprocess_threads= 16image, Label=Tf.train.shuffle_batch ([Images,labels], batch_size=batch_size, Num_threads=num_preprocess_threads, Min_after_dequeue= 20000, capacity = 20192)                                returnimage, Tf.reshape (label, [Batch_size])

3) If you want to read the data is a picture, or other types of format, you can first convert the data into TensorFlow standard support format tfrecords, it is actually a binary file, by modifying the tf.train.Example features, will Protocol buffer is serialized as a string, and then through Tf.python_io. Tfrecordwriter writes the serialized string to Tfrecords and then reads the tfrecords in the same way as above, except that the reader becomes TF. Tfrecordreader, followed by a parser tf.parse_single_example, and then decoded with the decoder Tf.decode_raw.

For example, for a generated anti-network GAN, I used this form to enter, some of the code is as follows:

def_int64_feature (value):returnTf.train.Feature (int64_list = tf.train.Int64List (value =[value]))def_bytes_feature (value):returnTf.train.Feature (bytes_list = tf.train.BytesList (value =[value])) defconvert_to (Data_path, name):"""converts S dataset to Tfrecords"""rows= 64cols= 64Depth=DEPTH forIiinchRange (12): Writer= Tf.python_io. Tfrecordwriter (name + str (ii) +'. Tfrecords')         forImg_nameinchOs.listdir (Data_path) [ii*16384: (ii+1) *16384]: Img_path= Data_path +img_name img=Image.open (Img_path) H, W= Img.size[:2] J, K= (h-output_size)/2, (w-output_size)/2Box= (J, K, J + output_size, + Koutput_size) img= Img.crop (box =box) img=img.resize ((rows,cols)) Img_raw=img.tobytes () example= Tf.train.Example (features = tf.train.Features (feature = {                                    'Height': _int64_feature (rows),'Weight': _int64_feature (cols),'Depth': _int64_feature (depth),'Image_raw': _bytes_feature (Img_raw)})) Writer.write (example. Serializetostring ()) Writer.close ()defRead_and_decode (filename_queue):"""Read and decode Tfrecords"""    #filename_queue = Tf.train.string_input_producer ([filename_queue])Reader =TF. Tfrecordreader () _, Serialized_example=Reader.read (filename_queue) features= Tf.parse_single_example (Serialized_example,features = {                        'Image_raw': TF. Fixedlenfeature ([], tf.string)}) Image= Tf.decode_raw (features['Image_raw'], tf.uint8)returnImage

Here, my data_path below there is a 16384*12 map, through 12 write example operation, the picture data into 12 tfrecords, each tfrecords inside there are 16384 pictures.

4) If you want to define your own read Data operation, please refer to https://www. tensorflow.org/how_tos/new_data_formats/.

OK, today's car station, please take good carry-on items ready to get off, tomorrow, the old driver also has a bus, please remember to ride on time, the car waits for no one.

Reference documents:

1. https://www. tensorflow.org/how_tos/

2. No more.

TF Boys (TensorFlow Boys) nurturance (II.)