Classic convolutional neural network structure--lenet-5, AlexNet, VGG-16

The structure of the classic convolutional neural network generally satisfies the following expressions:

Output layer, (convolutional layer +--pooling layer?) ) +-Full connection layer +

In the above formula, "+" means one or more, "? "represents one or 0, such as" convolutional layer + ", which represents one or more convolutional layers," pooling layer? " "represents one or 0 pooled layers. "--" indicates the forward direction.

The LeNet-5, AlexNet, and VGG-16 structures are described separately below.

1. LeNet-5 (modern)

Figure 1 LeNet-5

1.1 LENET-5 Structure:

• Input layer

The image size is 32x32x1, where 1 is a black-and-white image with only one channel.

• Convolution layer

Filter size 5x5,filter Depth (number) is 6,padding 0, the convolution step $s =1$, the output matrix size is 28x28x6, where 6 represents the number of filter.

• Pooling Layer

Average pooling,filter size 2x2 (i.e. $f =2$), step $s =2$,no padding, output matrix size is 14x14x6.

• Convolution layer

Filter size 5x5,filter number is 16,padding to 0, convolution step $s =1$, the output matrix size is 10x10x16, where 16 represents the number of filter.

• Pooling Layer

Average pooling,filter size 2x2 (i.e. $f =2$), step $s =2$,no padding, output matrix size is 5x5x16. Note that at the end of the layer, you need to flatten the 5x5x16 matrix into a 400-dimensional vector.

• Fully connected layer (Fully Connected LAYER,FC)

The number of neuron is 120.

• Fully connected layer (Fully Connected LAYER,FC)

The number of neuron is 84.

• Full connection layer, output layer

Now the version of the LENET-5 output layer will generally take the Softmax activation function, the LeNet-5 presented in the paper used in the activation function is not softmax, but it is not used now. The number of neurons in this layer is 10, which represents 0~9 10 numeric categories. (Figure 1 does not actually draw a box that represents the full join layer, but directly uses the $\hat{y}$ to represent the output layer.) ）

1.2 LeNet-5 Some properties:

• If the input layer does not count the number of layers in the neural network, then LeNet-5 is a 7-layer network. (Some places may also consider convolution and pooling as a layer)
• The LeNet-5 has approximately 60,000 parameters.
• As the network grows deeper, the height and width of the image are shrinking, while the number of channel numbers in the image has been increasing.
• Now commonly used in the LENET-5 structure and Yann LeCun professor in the 1998 paper, the structure in some places, such as the use of activation function, now generally use ReLU as an activation function, the output layer is generally selected softmax.

2. AlexNet

Figure 2 AlexNet

2.1 AlexNet Structure:

• Input layer: The image size is 227x227x3, where 3 indicates the channel number (R,G,B) of the input image is 3.
• convolution layer : Filter size 11x11,filter Number 96, convolution step $s = 4$. (The filter size lists only the width and height, the channel number of the filter matrix is the same as the channel number of the input picture, not listed here)
• Pooled layer: Max pooling,filter size 3x3, step $s = 2$.
• convolution layer : Filter size 5x5,filter number 256, step $s = 1$,padding use same convolution, even if the convolution output image and the input image remain the same width and height.
• Pooled layer: Max pooling,filter size 3x3, step $s = 2$.
• convolution layer : Filter size 3x3,filter number 384, step $s = 1$,padding use same convolution.
• convolution layer : Filter size 3x3,filter number 384, step $s = 1$,padding use same convolution.
• convolution layer : Filter size 3x3,filter number 256, step $s = 1$,padding use same convolution.
• Pooled layer: Max pooling,filter size 3x3, step $s = 2$; After the pooling operation is finished, the output matrix of size 6x6x256 is flatten to a 9216-dimensional vector.
• Fully connected layer: The number of neuron is 4096.
• Fully connected layer: The number of neuron is 4096.
• Full connection layer, output layer: Softmax activation function, neuron number is 1000, representing 1000 categories.

2.2 AlexNet Some properties:

• Approx. 60million parameters;
• Use ReLU as the activation function.

3. VGG-16

Figure 3 VGG-16

3.1 VGG-16 Structure:

• Input layer
• Convolution layer
• Convolution layer
• Pooling Layer
• Convolution layer
• Convolution layer
• Pooling Layer
• Convolution layer
• Convolution layer
• Convolution layer
• Pooling Layer
• Convolution layer
• Convolution layer
• Convolution layer
• Pooling Layer
• Convolution layer
• Convolution layer
• Convolution layer
• Pooling Layer
• Fully connected Layer
• Fully connected Layer
• Full connection layer, output layer

3.2 VGG-16 Some properties:

• The 16 in VGG-16 indicates that there are 16 layers in the entire network with trainable parameters. (the trainable parameter refers to a parameter that can be updated by back-propagation)
• The VGG-16 has approximately 138million parameters.
• All convolutional layers in the VGG-16 filter width and height are 3, and the step size is 1,padding using same convolution; All pooled layers have a filter width and height of 2, and the step size is 2.

References

Course 4 convolutional neural Networks by Anderw Ng

"TensorFlow Google deep Learning framework"