LeNet-5 model and its TensorFlow code implementation

This paper first analyzes the structure of the LENET-5 model, and then based on the LENET-5 model to write the TensorFlow code to achieve mnist digital recognition, the code part of the detailed annotation, at present also in the learning phase, there are errors welcome to point out that we learn together.

The LENET-5 model structure diagram is as follows:

The LENET-5 model has a total of 7 layers.

First layer: convolution layer

The first layer of the convolution layer input is the original image, the original image size is 32x32x1. The filter size of the convolution layer is 5x5, the depth is 6, the total 0 is not used, and the step is 1. Since no 0 supplements are used, the output of this layer is 32-5+1=28, with a depth of 6. This convolution layer has a total of 5x5x1x6+6=156 parameters, of which 6 is the number of offset parameters, the number of parameters of the convolution layer only and the size of the filter, depth and the depth of the current layer node matrix. Because the next layer node matrix has 28x28x6=4704 nodes, each node is connected with the 5x5=25 current layer node, so the convolution layer of this layer has 4704x (25+1) = 122,304 connections altogether.

Second layer: Pool layer

The input of this layer is the output of the first layer, which is a 28x28x6 node matrix. This layer adopts the filter size is 2x2, step is 2, so the output matrix size of this layer is 14x14x6.

Third layer: convolution layer

The input matrix size of this layer is 14x14x6, the filter size is 5x5, the depth is 16, the total 0 is not used, and the step is 1. The output of this layer is 14-5+1=10, the depth is 16, the output matrix size is 10x10x16. This layer parameter has 5x5x6x16+16=2416, the connection has 10x10x16x (5x5+1) = 41,600.

Layer Fourth: Pool layer

The input matrix size of this layer is 10x10x16, the filter size is 2x2, the step is 2, the output matrix of this layer is 5x5x16.

Fifth Floor: Full connection layer

The input matrix size of this layer is 5x5x16, this layer is called the convolution layer in the LENET-5 model, but because the size of the filter is 5x5, it is not different from the whole connection layer, which is directly regarded as the full connection layer. This layer is entered as a 5x5x16 matrix, which is pulled straight into a vector of length 5x5x16, and a three-dimensional matrix is pulled until one dimensional space is represented as a vector so that it can be trained in the full join layer. The number of output nodes in this layer is 120, so there is a total of 5x5x16x120+120=48120 parameters.

Sixth Floor: Full connection layer

The number of input nodes in this layer is 120, the number of output nodes is 84, a total of 120x84+84=10164 parameters.

Seventh Floor: Full connection layer

The number of input nodes in this layer is 84, the number of output nodes is 10, a total of 84x10+10=850 parameters.

Next, a mnist digital recognition code based on the LENET-5 model is presented in TensorFlow code.

mnist DataSet PNG Picture format download Link: Link: http://pan.baidu.com/s/1eRG6TqU Password: uf4m

TensorFlow Version: 1.2.1

The code is as follows:

From skimage import io,transform import OS import glob import numpy as NP import TensorFlow as TF #将所有的图片重新设置尺寸为32 *32 W = h = c = 1 #mnist数据集中训练数据和测试数据保存地址 train_path = "e:/data/datasets/mnist/train/" Test_path = "E:/data/datasets/mni
    st/test/"#读取图片及其标签函数 def read_image (path): Label_dir = [path+x for x in Os.listdir (path) if Os.path.isdir (path+x)] images = [] labels = [] for Index,folder in Enumerate (LABEL_DIR): for IMG in Glob.glob (folder+ '/*.png ') : Print ("Reading the image:%s"%img) image = Io.imread (img) image = Transform.resize (im Age, (W,h,c)) images.append (image) Labels.append (index) return Np.asarray (images,dtype=np.float Np.asarray (labels,dtype=np.int32) #读取训练数据及测试数据 Train_data,train_label = Read_image (Train_path) Test_data, Test_label = Read_image (test_path) #打乱训练数据及测试数据 train_image_num = Len (train_data) Train_image_index = Np.arange (train_ Image_num) NP.RANDOM.SHUFFLE (train_image_index) train_data = Train_data[train_image_index] Train_label = Train_label[train_image_index] Test_ Image_num = Len (test_data) Test_image_index = Np.arange (test_image_num) np.random.shuffle (test_image_index) test_data = Test_data[test_image_index] Test_label = Test_label[test_image_index] #搭建CNN x = Tf.placeholder (tf.float32,[None,w,h , c],name= ' x ') Y_ = Tf.placeholder (tf.int32,[none],name= ' Y_ ') def inference (Input_tensor,train,regularizer): #第一层: Convolution layer
    , the filter size is 5x5, the depth is 6, does not use all 0 supplements, the step size is 1. #尺寸变化: 32x32x1->28x28x6 with Tf.variable_scope (' Layer1-conv1 '): conv1_weights = tf.get_variable (' Weight ', [5, 5,c,6],initializer=tf.truncated_normal_initializer (stddev=0.1)) conv1_biases = tf.get_variable (' bias ', [6],initial Izer=tf.constant_initializer (0.0)) Conv1 = tf.nn.conv2d (input_tensor,conv1_weights,strides=[1,1,1,1],padding= ' VAL
    ID ') RELU1 = Tf.nn.relu (Tf.nn.bias_add (conv1,conv1_biases)) #第二层: The pool layer, the size of the filter is 2x2, with a total of 0 supplements, step size is 2. #尺寸变化: 28x28x6-> 14x14x6 with Tf.name_scope (' Layer2-pool1 '): Pool1 = Tf.nn.max_pool (relu1,ksize=[1,2,2,1],strides=[1,2,2,1],pa
    dding= ' SAME ') #第三层: The convolution layer, the size of the filter is 5x5, the depth is 16, not using all 0 supplements, step 1. #尺寸变化: 14x14x6->10x10x16 with Tf.variable_scope (' Layer3-conv2 '): conv2_weights = tf.get_variable (' Weight ', [5 , 5,6,16],initializer=tf.truncated_normal_initializer (stddev=0.1)) conv2_biases = tf.get_variable (' bias ', [16],init 
        Ializer=tf.constant_initializer (0.0)) Conv2 = tf.nn.conv2d (pool1,conv2_weights,strides=[1,1,1,1],padding= ' VALID ')
    RELU2 = Tf.nn.relu (Tf.nn.bias_add (conv2,conv2_biases)) #第四层: The pool layer, the size of the filter is 2x2, the use of all 0 supplements, step size of 2. #尺寸变化: 10x10x6->5x5x16 with Tf.variable_scope (' Layer4-pool2 '): Pool2 = Tf.nn.max_pool (relu2,ksize=[1,2,2,1), strides=[1,2,2,1],padding= ' SAME ') #将第四层池化层的输出转化为第五层全连接层的输入格式.
    The fourth layer output is the 5x5x16 matrix, however the fifth layer full join layer needs the input format #为向量, so we need to pull the 5x5x16 matrix representing each picture into a vector of 5x5x16 length. #举例说, 64 pictures are trained at a time, then the size of the fourth layer of the pool is (64,5,5,16), which is a vector, nodes=5x5x16=400, dimension size changed to (64,400) Pool_shape = Pool2.get_shape (). As_list () nodes = pool_shape[1]*pool_shape[2]*pool_shape[3] reshaped = Tf.reshape (Pool2,[-1,nodes]) #第五层: Fully connected layer, nodes=5x5x16=400,400->120 full connection #尺寸变化: For example, a group of training samples is 64, then the size changes to 64x40
    0->64x120 #训练时, the introduction of dropout,dropout in training will randomly change the output of some nodes to 0,dropout can avoid the problem of fitting.
    #这和模型越简单越不容易过拟合思想一致, and regularization limit weights, so that the model can not be arbitrarily fitted to the random noise in training data, so as to avoid the consensus of the fitting idea.
    #本文最后训练时没有采用dropout, the dropout entry parameters are set to false because the training and testing are written together without separation, but you can try. With Tf.variable_scope (' Layer5-fc1 '): fc1_weights = tf.get_variable (' Weight ', [nodes,120],initializer=tf.truncated_ Normal_initializer (stddev=0.1)) if Regularizer!= None:tf.add_to_collection (' Losses ', Regularizer (fc1_ weights)) fc1_biases = tf.get_variable (' bias ', [120],initializer=tf.constant_initializer (0.1)) FC1 = Tf.nn. Relu (Tf.matmul (reshaped,fc1_weights) + fc1_biases) If train:fc1 = Tf.nn.dropout (fc1,0.5) #第六层: Full Company Connecting layer, 120->84 full connection #尺寸变化: For example, a group of training samples for 64, then the rulerInch change for 64x120->64x84 with Tf.variable_scope (' LAYER6-FC2 '): fc2_weights = tf.get_variable (' Weight ', [120,84],ini Tializer=tf.truncated_normal_initializer (stddev=0.1)) if Regularizer!= None:tf.add_to_collection (' Lo SSEs ', Regularizer (fc2_weights)) fc2_biases = tf.get_variable (' bias ', [84],initializer=tf.truncated_normal_initiali Zer (stddev=0.1)) FC2 = Tf.nn.relu (Tf.matmul (fc1,fc2_weights) + fc2_biases) If TRAIN:FC2 = tf. Nn.dropout (fc2,0.5) #第七层: Fully connected layer (approximate representation), 84->10 full connection #尺寸变化: For example, a group of training samples is 64, then the size changes to 64x84->64x10.
    Finally, the 64x10 matrix Softmax the probability of 64 images sorted into each number, #即得到最后的分类结果. With Tf.variable_scope (' LAYER7-FC3 '): fc3_weights = tf.get_variable (' Weight ', [84,10],initializer=tf.truncated_norm Al_initializer (stddev=0.1)) if Regularizer!= None:tf.add_to_collection (' Losses ', Regularizer (Fc3_weig HTS)) fc3_biases = tf.get_variable (' bias ', [10],initializer=tf.truncated_normal_initializer (StdDev=0.1)) Logit = Tf.matmul (fc2,fc3_weights) + fc3_biases return logit #正则化, cross entropy, mean cross entropy, loss function, minimize loss function, forecast and actual equal comparison,
The Tf.equal function gets true or FALSE, #accuracy首先将tf. The boolean value obtained by equal is converted to float, i.e. true to 1.,false to 0, and finally to the average, i.e. the correct rate of a group of samples.
#比如: A group of 5 samples, tf.equal to [true false false], to the float type [1.0 1.0 0], and the accuracy rate is 2./5=40%. Regularizer = Tf.contrib.layers.l2_regularizer (0.001) y = inference (x,false,regularizer) cross_entropy = Tf.nn.sparse_ Softmax_cross_entropy_with_logits (logits=y,labels=y_) Cross_entropy_mean = Tf.reduce_mean (cross_entropy) loss =
Cross_entropy_mean + tf.add_n (tf.get_collection (' losses ')) Train_op = Tf.train.AdamOptimizer (0.001). Minimize (loss) Correct_prediction = Tf.equal (Tf.cast (Tf.argmax (y,1), Tf.int32), y_) accuracy = Tf.reduce_mean (Tf.cast (correct_ Prediction,tf.float32) #每次获取batch_size个样本进行训练或测试 def get_batch (data,label,batch_size): For Start_index in range (0,le N (data)-batch_size+1,batch_size): Slice_index = slice (start_index,start_index+batch_size) yield DAta[slice_index],label[slice_index] #创建Session会话 with TF. Session () as Sess: #初始化所有变量 (weights, biases, etc.) Sess.run (Tf.global_variables_initializer ()) #将所有样本训练10次, each training in 64 for a group of all
    Sample.
    #train_num可以设置大一些.
        Train_num = Batch_size = The For I in Range (train_num): train_loss,train_acc,batch_num = 0, 0, 0 For Train_data_batch,train_label_batch in Get_batch (train_data,train_label,batch_size): _,ERR,ACC = SESS.R Un ([train_op,loss,accuracy],feed_dict={x:train_data_batch,y_:train_label_batch}) Train_loss+=err;train_acc+=ac

        C;batch_num+=1 Print ("Train loss:", train_loss/batch_num) print ("Train acc:", Train_acc/batch_num) Test_loss,test_acc,batch_num = 0, 0, 0 for Test_data_batch,test_label_batch in Get_batch (TEST_DATA,TEST_LABEL,BATC H_size): ERR,ACC = Sess.run ([Loss,accuracy],feed_dict={x:test_data_batch,y_:test_label_batch}) TES T_loss+=err;test_acc+=acc;batch_num+=1 Print ("tesT loss: ", test_loss/batch_num) print (" Test acc: ", Test_acc/batch_num) 

The final test accuracy rate is about 99%.

A random run of training and testing results are as follows:

...
Test loss:0.0766476487789
Test acc:0.98577724359
train loss:0.0529618164884
train acc:0.993930096051
test loss:0.0702898033345
test acc:0.987680288462
train loss:0.0511233446804
train ACC: 0.994213580576
test loss:0.0881681445843
test acc:0.982672275641
train loss:0.0503073274342
Train acc:0.994030149413
test loss:0.0700190923033
Test acc:0.986979166667

Reference: "TensorFlow: Practical Google Depth Learning framework"