Deep learning Note (i) convolutional neural network (convolutional neural Networks)

I. Convolution

convolutional Neural Networks (convolutional neural Networks) are neural networks that share parameters spatially. Multiply by using a number of layers of convolution, rather than a matrix of layers. In the process of image processing, each picture can be regarded as a "pancake", which includes the height of the picture, width and depth (that is, color, in RGB).

Without changing the weight of the case, the above with K output Small network corresponding small block slide through the entire image, you can get a width, height, and depth is different new images.

Convolution when there are many ways to fill the image, the following main introduction of two, one is the same fill , one is a valid fill .

As shown in the Purple box, the left is a valid fill and the right is the same fill. In the same fill, the part that goes out of bounds uses a 0-plus approach, making the input and output images the same size. In a valid population, the method of supplementing 0 is not used, and the boundary cannot be exceeded, so the size of the input is larger than the size of the output.

Displays the size of the resulting output image with a 3x3 grid on the 28x28 image using different step sizes and fill methods:

The following is an understanding of the convolution process with two dynamic graphs:

The first is a convolution process that is effectively populated with a 3x3 grid on a 5x5 image:

The second is the convolution process with the same padding on the 5x5 image using a 3x3 grid, moving in the following way: http://cs231n.github.io/convolutional-networks/

Reviewing the whole process is to increase the depth of the network one layer at a time, eventually getting a deep and narrow representation, then connecting it to the fully connected layer and then training the classifier.

Second, local connection and weight sharing

In general, local connection and weight sharing are the methods to reduce the parameters, making feature extraction more effective.

The left half is an example of a fully connected neural network. The picture is a 1000x1000 image, the next hidden layer has $10^6$ neurons, then there will be 1000x1000x$10^6$=$10^{12}$ parameters.

The second half is an example of a locally connected neural network. The figure is still a 1000x1000 image, the next hidden layer has $10^6$ neurons, but using a 10x10 convolution core, connected to the 10x10 of the local image, then there will be 10x10x$10^6$=$10^8$ parameters.

In practical application, there are some special cases, which involve the problem of statistical invariance. For example, we want to identify the animal category in the image, then the position of the animal in the picture (upper-left, middle or lower-right corner) is not important, which is called translational invariance; for example, in the process of recognizing numbers, the color of the numbers does not affect the results, or, in language processing, The position of some words in a sentence does not affect the meaning of their representation. When both inputs can get the same information, we should share weights and use these inputs to train weights together.

In the left half, there is an example of a locally connected neural network that is not shared with weights.

In the right half of the section, weight sharing is used. The figure is a 1000x1000 image, with 100 10x10 of convolution cores, which will eventually have 100x10x10=10k parameters. Using local joins and weight sharing greatly reduces the number of parameters. The sharing weights make the problem of statistical invariance be solved effectively.

Third, pooling

After convolution, in order to introduce invariance, at the same time to prevent over-fitting problems or less-fitting problems, reduce the computational volume, we often do pool-based processing. The pooling process is as shown. Therefore, after pooling, the width and height of the image will usually change to 1/2 of the original.

This includes three pooling methods for Max pooling, Mean pooling, and stochastic pooling.

Two more commonly used are: Max pooling is the maximum value that is selected within the kernel range, and Mean pooling is the average of the selected kernel range.

Analysis of convolution and pooling in LeNet-5

In LeNet-5, the input layer is the size of 32x32.

In the first convolution, 6 convolution cores were used, and the feature map of C1:6 28x28 was obtained.

And then the next sampling, get S2: Feature map width, high to the original 1/2, that is, 28/2=14, feature map size into 14x14, the number of feature maps is unchanged.

Then the second convolution, using 16 convolution cores, obtained the feature map of C3:16 Zhang 10x10.

Then the next sampling, get S4: The feature map width, high to the original 1/2, that is, the 10/2=5, the feature map size into 5x5, the number of feature map is unchanged.

After entering the convolution layer c5,120 Zhang 1x1 full connection feature map, and S4 full connection.

The pictures and contents of this article are reference or from the following information:

[1] Udacity's Deep learning course: https://cn.udacity.com/course/deep-learning--ud730

[2] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks[j]. Advances in neural information processing Systems, 2012, 25 (2): 2012.

[3] LeCun y, Bottou L, Bengio y, et al gradient-based learning applied to document RECOGNITION[J]. Proceedings of the IEEE, 1998, 86 (11): 2278-2324.

[5]http://www.jeyzhang.com/cnn-learning-notes-1.html

[6]http://blog.csdn.net/stdcoutzyx/article/details/41596663

[7] Cs231n:convolutional Neural Networks for Visual recognition

[8]http://ibillxia.github.io/blog/2013/04/06/convolutional-neural-networks/

This article is a personal study note, the level is limited, if there are omissions, please point out, thank you.

Deep learning Note (i) convolutional neural network (convolutional neural Networks)