TensorFlow Example: (Convolution neural network) LENET-5 model

There are infinitely many neural networks which can be obtained by any combination of the convolution layer, the pool layer and so on, and what kind of neural network is more likely to solve the real image processing problem. In this paper, a general model of convolution neural network structure design is given through LeNet-5 model. LeNet-5 Model

The LENET-5 model, presented by Professor Yann LeCun in his paper gradient-based learning to document applied in 1998, was the first convolution neural network successfully applied to the problem of digital recognition. The LENET-5 model has a total of 7 layers, and the following illustration shows the architecture of the LENET-5 model:
first layer, convolution layer

This layer of input is the original image pixel 32*32*1. The first convolution layer filter size is 5*5, the depth is 6, does not use full 0 padding, the step size is 1. So the output of this layer: 28*28*6, the convolution layer has a 5*5*1*6+6=156 of the second layer of parameters, the pool layer

The input of this layer is the output of the first layer, which is a 28*28*6 node matrix. The size of the filter used in this layer is 2*2, the length and width of the step are 2, so the output matrix of this layer is 14*14*6. third layer, convolution layer

The input matrix size of this layer is 14*14*6, the filter size used is 5*5 and the depth is 16. This layer does not use full 0 padding, and the step is 1. The output matrix size of this layer is 10*10*16. This layer has 5*5*6*16+16=2416 parameters. fourth floor, pool layer

The input matrix size of this layer is 10*10*16. The size of the filter used in this layer is 2*2, the length and width of the step are 2, so the output matrix of this layer is 5*5*16. fifth Floor, fully connected layer

The input matrix size of this layer is 5*5*16, in LeNet-5 paper, this layer becomes the convolution layer, but because the size of the filter is 5*5, it is no different from the whole connection layer. If the nodes in the 5*5*16 matrix are pulled into a vector, then this layer is the same as the full connection layer. The number of output nodes in this layer is 120, with a total of 5*5*16*120+120=48120 parameters. Sixth Floor, fully connected layer

The number of input nodes in this layer is 120, the number of output nodes is 84, the total parameter is 120*84+84=10164. seventh floor, fully connected layer

The number of input nodes in this layer is 84, the number of output nodes is 10, and the total parameters are 84*10+10=850 tensorflow implementation LeNet-5

The following is a TensorFlow program to implement a convolution neural network similar to the LENET-5 model to solve the problem of mnist digital recognition. lenet_inference.py

# _*_ Coding:utf-8 _*_ import tensorflow as TF # Configure parameters for neural Networks Input_node = 784 Output_node = Ten image_size = Num_channe LS = 1 Num_labels = 10 # The size and depth of the first convolution layer conv1_deep = Conv1_size = 5 # The size and depth of the second convolution conv2_deep = Conv2_size = 5 # Fully connected layer The number of nodes Fc_size = 512 # Defines the forward propagation process of the convolution neural network. A new parameter train is added here to differentiate between the training process and the test process. The dropout method will be used in this program # dropout can further enhance the reliability of the model and prevent the fitting (dropout process only used during training) def inference (Input_tensor, Train, Regularizer): Wit H Tf.variable_scope (' Layer1-conv1 '): conv1_weights = tf.get_variable (' Weight ', [Conv1_size, Conv1_size, NUM_CHANNE
        LS, Conv1_deep], Initializer=tf.truncated_normal_initializer (stddev=0.1)) conv1_biases = tf.get_variable (' bias ', [conv1_deep], Initializer=tf.constant_initia Lizer (0.0)) Conv1 = tf.nn.conv2d (Input_tensor, Conv1_weights, strides=[1, 1, 1, 1], padding= ' SAME ') relu1 = Tf.nn.relu (Tf.nn.bias_add (CONV1, conv1_biases)) with Tf.name_scope (' LAYer2-pool1 '): Pool1 = Tf.nn.max_pool (RELU1, Ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding= ' SAME ') with TF . Variable_scope (' Layer3-conv2 '): conv2_weights = tf.get_variable (' Weight ', [Conv2_size, Conv2_size, Conv1_deep, CO Nv2_deep], Initializer=tf.truncated_normal_initializer (stddev=0.1)) conv2_ biases = tf.get_variable (' bias ', [conv2_deep], Initializer=tf.constant_initializer ( 0.0)) Conv2 = tf.nn.conv2d (Pool1,conv2_weights, strides=[1, 1, 1, 1], padding= ' SAME ') RELU2 = Tf.nn.relu (t F.nn.bias_add (Conv2, conv2_biases)) with Tf.name_scope (' Layer4-pool2 '): Pool2 = Tf.nn.max_pool (RELU2, ksize=[ 1, 2, 2, 1], strides=[1, 2, 2, 1], padding= ' SAME ') Pool2_shape = Pool2.get_shape (). As_list () nodes = pool2_shape[ 1] * pool2_shape[2] * pool2_shape[3] reshaped = Tf.reshape (Pool2, [pool2_shape[0], nodes]) with TF.VARIABLE_SCOP
       E (' Layer5-fc1 '): Fc1_weights = tf.get_variable (' weight ', [nodes, Fc_size], initializer=tf.truncated_ Normal_initializer (stddev=0.1)) if Regularizer!= None:tf.add_to_collection (' Losses ', Regularizer (FC1 _weights)) fc1_biases = tf.get_variable (' bias ', [fc_size], Initializer=tf.con
            Stant_initializer (0.0)) FC1 = Tf.nn.relu (Tf.matmul (reshaped, fc1_weights) + fc1_biases) if train: FC1 = Tf.nn.dropout (FC1, 0.5) with Tf.variable_scope (' LAYER6-FC2 '): fc2_weights = tf.get_variable (' Weight
        ', [Fc_size, Num_labels], Initializer=tf.truncated_normal_initializer (stddev=0.1)) If Regularizer!= None:tf.add_to_collection (' Losses ', Regularizer (fc2_weights)) fc2_biases = t
        F.get_variable (' bias ', [num_labels], Initializer=tf.constant_initializer (0.0)) Logit = Tf.matmul (FC1, fc2_weights) + fc2_biases return logit 

lenet_train.py

# _*_ coding:utf-8 _*_ import os import tensorflow as TF import NumPy as NP from Tensorflow.examples.tutorials.mnist Imp ORT Input_data # Load constants and forward-propagating functions defined in mnist_inference.py import lenet_inference # Configure parameters for neural Networks batch_size = Learning_rate_ba SE = 0.8 Learning_rate_decay = 0.99 Regularaztion_rate = 0.0001 Train_steps = 30000 Moving_average_decay = 0.99 MODEL_SAV E_path = "./model/" model_name = "MODEL3.CKPT" Def Train (mnist): x = Tf.placeholder (Tf.float32, [Batch_size, lenet_in Ference. Image_size, Lenet_inference. Image_size, Lenet_inference. Num_channels], name= ' x-input ') Y_ = Tf.placeholder (Tf.float32, [None, Lenet_inference. Output_node], name= ' y-input ') Regularizer = Tf.contrib.layers.l2_regularizer (regularaztion_rate) y = Lenet_infer Ence.inference (x, Train, regularizer) Global_step = tf. Variable (0, trainable=false) variable_average = Tf.train.ExponentialMovingAverage (moving_average_deCay, global_step) Variable_average_op = variable_average.apply (Tf.trainable_variables ()) Cross_entropy = Tf.nn.sparse_softmax_cross_entropy_with_logits (Labels=tf.argmax (Y_, 1), logits=y) Cross_entropy_mean = Tf.reduce_ Mean (cross_entropy) loss = Cross_entropy_mean + tf.add_n (tf.get_collection (' losses ')) Learning_rate = TF.TRAIN.E Xponential_decay (Learning_rate_base, Global_step=global_step, decay_steps=m Nist.train.num_examples/batch_size, Decay_rate=learning_rate_decay) t Rain_step = Tf.train.GradientDescentOptimizer (learning_rate). Minimize (loss, global_step=global_step) with Tf.control_dependencies ([Train_step, Variable_average_op]): Train_op = Tf.no_op (name= ' train ') saver = Tf.trai N.saver () with TF. Session () as Sess:tf.global_variables_initializer (). Run () to I in Range (train_steps): XS, y s = mnist.train.neXt_batch (batch_size) xs = np.reshape (XS, [Batch_size, Lenet_inference. Image_size, Lenet_inference. Image_size, Lenet_inference.
            Num_channels] _, loss_value, step = Sess.run ([Train_op, Loss, Global_step], Feed_dict={x:xs, Y_: ys}) If I% 1000 = 0:print ("After%d training steps, loss on training" "Batch is %g "% (step, Loss_value)) Saver.save (Sess, Os.path.join (Model_save_path, model_name), global_step=global_s Tep def Main (argv=none): Mnist = input_data.read_data_sets ("e:\ Research \tensorflow Course \mnist_data", one_hot=true) train ( mnist) If __name__ = = ' __main__ ': Tf.app.run ()

How to design the convolution neural network architecture

The following regularization formula summarizes some of the classic convolution neural network architectures for picture classification problems:
input layer → (convolution layer +→ pool layer.) ) +→ Full connection layer +

"+" means one or more layers, ". "Means yes or no
In addition to the LENET-5 model, the first alexnet model of the 2012 Imagenet ILSVRC Image Classification Challenge, the 2013 ILSVRC first ZF NET model has been in place for 2014 years the second Vggnet model architecture satisfies the above regular expression. How to set the size of the volume or pool layer configuration filter: 1 or 3 or 5, some networks have 7 or even 11 of the depth of the filter: Increment by layer. After each layer of the pool, the depth of the convolution layer * 2 convolution layer step: Generally 1, some will use 2 or even 3 pool layer: The most is max_pooling, filter side length is generally 2 or 3, the step is generally 2 or 3