TensorFlow realization of convolution neural network (Advanced) _ Neural network

If you use 100k batch in this model, and combine the decay of learning rate (that is, the rate of learning is reduced by a ratio every once in a while), the correct rate can be as high as 86%. There are about 1 million parameters to be trained in the model, and the total amount of arithmetic to be estimated is about 20 million times. So this convolution neural network model, using some techniques.
(1) Regularization of the L2 of the weight.
(2) The image is flipped, random shearing and other data enhancement, manufacturing more samples.
(3) Use the LRN layer behind each convolution-maximum pool layer to enhance the generalization capability of the model.

The combination of convolution and pooling is now a standard component for image recognition. The convolution layer mainly makes the feature extraction, the whole connection layer starts the multiple features to match, and classifies. The training of the convolution layer is more complicated than the whole connection layer, and the training of the whole connection layer is the multiplication of some matrices.

Download the TENSORFOW model, which uses the class (cifar10.py and cifar10_input.py) to read CTFAR-10 data (CTFAR-10 a classic dataset) when building models

git clone git@github.com:tensorflow/models.git

Convolution neural network structure:
CONV1 convolution layer and activation function
Maximum pool of pool1
Norm1 LRN
Conv2 convolution layer and activation function
Norm2 LRN
Maximum pool layer of pool2
LOCAL3 full join layer and activation function
LOCAL4 full join layer and activation function
Output results of Logits model inference

# coding:utf-8 # Loads Common libraries, numpy time, and loads classes that automatically download, read CIFAR-10 data in tlensorfow models. Import cifar10,cifar10_input import TensorFlow as TF import numpy as NP import time ####### #输入数据 ######## # Training number, batch size (3
000 batch, each batch contains 128 samples). Max_steps = 3000 batch_size = 128 # Download the default path for CIFAR-10 data Data_dir = '/tmp/cifar10_data/cifar-10-batches-bin ' ####### #初始化权重 # #
###### # defines the function that initializes the weight, and still initializes the weights using the normal distribution of the tf.truncated_normal truncation. # here gives weight a L2 loss, which is equivalent to doing a L2 regularization process. This collection, named "Losses", will be used when the overall loss is computed, with Def variable_with_weight_loss (Shape, StdDev, wl): var = tf. Variable (Tf.truncated_normal (shape, StdDev = StdDev)) If WL is not None:weight_loss = tf.multiply (Tf.nn.l2_lo SS (Var), wl, name = ' Weight_loss ') tf.add_to_collection (' losses ', Weight_loss) return to Var ####### #数据处理 ####### # Extract the Cifar10 data into Data_dir, then annotate the next line of code, cancel run # (used to cifar-10.py) to download the dataset using CIFAR-10, and extract the unwind to its default location Cifar10.maybe_download_ And_extract () # uses the Distorted_input function in the Cifar10_input class to produce the data that the training needs to be used, returns the encapsulated tensor, and each execution generates aA sample of the number of batch_size. Images_train, Labels_train = cifar10_input.distorted_inputs (Data_dir = data_dir, batch_size = batch_size) # using cifar10_in The Put.inputs function generates test data.
You need to crop the 24*24 blocks in the middle of the picture, and standardize the data operations. Images_test, labels_test = cifar10_input.inputs (Eval_data = True, Data_dir = data_dir, batch_size = batch_size) # Create input data Placeholder.
Batche_size is used when defining the network structure, so the first value of the data dimension needs to be set ahead of time.

Image_holder = Tf.placeholder (Tf.float32, [Batch_size, A, 3]) Label_holder = Tf.placeholder (Tf.int32, [batch_size]) ####### #设计网络结构 ######## # The first convolution layer # creates and initializes the convolution kernel, not the weight of the first convolution layer L2 the regular weight1 = variable_with_weight_loss (Shape = [ 5,5,3,64], StdDev = 5e-2, wl = 0.0) # convolution operation of input data Kernel1 = tf.nn.conv2d (Image_holder, weight1, [1,1,1,1], padding = ' SAME # This layer of bias is all initialized to 0, then the result of the convolution plus bias BIAS1 = tf. Variable (tf.constant (0.0, shape = [64])] # using activation function for non-linear conv1 = Tf.nn.relu (Tf.nn.bias_add (Kernel1, BIAS1)) # Use dimension 3*3 and step to 2*2 the largest pool layer processing data, the largest pool layer size and step inconsistent, increase the richness of the data Pool1 = Tf.nn.max_pool (conv1, ksize = [1,3,3,1], strides = [1,2,2,1], padding = ' SAME ') use LRN to process the results, create a competitive environment for the activities of local neurons, and enhance the generalization ability of the model Norm1 = TF.NN.LRN (pool1, 4, bias = 1.0, alpha = 0.001/9.0, b ETA = 0.75) # Second convolution layer (similar to the previous layer) # The volume kernel number on the previous layer is 64 (that is, 64 channels are output).
The number of input channels for the third dimension of the convolution kernel in this layer is 64. Weight2 = variable_with_weight_loss (shape = [5,5,64,64], StdDev = 5e-2, wl = 0.0) Kernel2 = tf.nn.conv2d (Norm1, Weight2, [
1,1,1,1], padding = ' SAME ') # bias values are all initialized to 0.1. BIAS2 = tf.
Variable (Tf.constant (0.1, shape = [])) Conv2 = Tf.nn.relu (Tf.nn.bias_add (Kernel2, BIAS2)) # Unlike the previous layer, LRN processing is performed at the maximum pool level. Norm2 = TF.NN.LRN (Conv2, 4, bias = 1.0, alpha = 0.001/9.0, beta = 0.75) pool2 = Tf.nn.max_pool (norm2, ksize = [1,3,3,1], S Trides = [1,2,2,1], padding = ' SAME ') # fully connected layer # The output from the previous layer is flatten.
The Tf.reshape function converts each sample into a one-dimensional vector.
Reshape = Tf.reshape (pool2, [Batch_size,-1]) # Gets the length of the data after flattening. Dim = Reshape.get_shape () [1].value # Initializes the weight of the full join layer, the number of hidden nodes is 384, and the standard deviation of the distribution is 0.04.
Sets a non-zero weight loss, all parameters of this pass are L2 regular constraints. WEIGHT3 = variable_with_weight_loss (Shape = [Dim, 384], StdDev = 0.04, wl = 0.004) # bias value initialized to 0.1 bias3 = tf. VariablE (Tf.constant (0.1, shape = [384])] # using activation function for non-linear local3 = Tf.nn.relu (Tf.matmul (reshape, WEIGHT3) + BIAS3) # Full-join layer (similar to previous layer) # The number of hidden layer nodes is reduced by only 192, and other super parameters remain constant WEIGHT4 = variable_with_weight_loss (shape = [384,192], StdDev = 0.04, wl = 0.004) Bias4 = TF . Variable (0.1, shape = [Tf.constant]) Local4 = Tf.nn.relu (Tf.matmul (Local3, WEIGHT4) + bias4) # output layer (put Softmax operation in Los
S part) # creates weight, whose normal distribution standard deviation is the reciprocal of the upper layer implied node, and does not count into the L2. WEIGHT5 = variable_with_weight_loss (shape = [192,10], StdDev = 1/192.0, wl = 0.0) BIAS5 = tf. Variable (tf.constant (0.0, shape = [ten])) # Softmax The reason for the following. 
We do not need to softmax the output of inference to obtain the final classification results. # Direct comparison of the inference output of various types of values can be. The calculation of Softmax is mainly to compute loss.
Therefore, the Softmax operation is integrated into the back appropriate. # model Inference Output logits = Tf.nn.relu (Tf.matmul (Local4, WEIGHT5) + bias5) ####### #计算CNN的loss ######## # Softmax and Cross ENT The calculations of the ropy loss are combined to get the final loss, which includes cross entropy loss and the weight L2 def loss (loss, logits) of the last two fully connected layers labels labels Abels, tf.int64) cross_entropy = Tf.nn.sparse_softmax_cross_entropy_with_Logits (logits = logits, labels = labels, name = ' Cross_entropy_per_example ') Cross_entropy_mean = Tf.reduce_mean (cross _entropy, name = ' cross_entropy ') tf.add_to_collection (' losses ', Cross_entropy_mean) return Tf.add_n (Tf.get_collec tion (' losses '), name = ' Total_loss ') # Loss The value passed in the function, get the final loss loss = loss (Logits, label_holder) ####### #训练设置 ######## # Select Optimizer, the learning rate is set to 1e-3 train_op = Tf.train.AdamOptimizer (1e-3). Minimize (loss) # The accuracy of top k in output results, that is, the accuracy of the class with the highest output score Top_k_op = Tf.nn.in_top_k (Logits, Label_holder, 1) # create default session Sess = TF. InteractiveSession () # initializes all model parameters Tf.global_variables_initializer (). Run () # launches the image data-enhanced thread queue, which is accelerated using 16 threads altogether. Do not start the following inference Tf.train.start_queue_runners () ####### #开始训练 ######## # Records the time each step takes, calculates every 10 step and displays the current loss,
The number of samples that can be trained per second, and the time spent in a batch. For step in Range (max_steps): Start_time = Time.time () # At each step of the training process, first get a batch data.
    The batch data is then passed into the train_op and loss calculations. Image_batch, Label_batch = Sess.run ([Images_train, Labels_train]) _, Loss_value = Sess.run ([Train_op, loss], feed_dict = {image_holder:image_batch, label_holder:label_batch}) Duratio n = time.time ()-start_time if step%10 ==0:examples_per_sec = batch_size/duration Sec_per_batch = Flo at (duration) Format_str = ("Step%d,loss=%.2f (%.1f example/sec;%.3f)") Print (sec/batch% (step, l Oss_value, Examples_per_sec, Sec_per_batch) # Test Set Evaluation Accuracy # Test Set sample Number Num_examples = 10000 Import Math # Calculate how many batch can evaluate all samples n um_iter = Int (Math.ceil (num_examples/batch_size)) True_count = 0 Total_sample_count = num_iter * Batch_size step = 0 # in
Each step uses the session's Run method to get the batch # of test and then executes the TOP_K_OP calculation model to predict the correct number of samples on top 1 of this batch.
# Finally, summarize all the correct results and get the correct quantity in all test samples. While step < Num_iter:image_batch, Label_batch = Sess.run ([images_test,labels_test]) predictions = Sess.run ([t OP_K_OP], feed_dict = {image_holder:image_batch, label_holder:label_batch}) True_count + + np.sum (predictions) St
EP + 1 # Finally, the accuracy evaluation results are calculated and printed out. Precision = true_count/total_sample_count # print (' precision @ 1 =%.3f '% precision) print (' num examples:%d num correc
 T:%d Precision @ 1:%0.02f '% (Total_sample_count, True_count, Precision))