Data Mining series (9)-an implementation of convolution neural network algorithms

Preface

It took a month to understand the convolutional neural network and implement it. However, it is still difficult for CNN to understand the convolutional neural network, instead of reading any blog or one or two papers, you can understand it. Instead, you can study it on your own and read the references at the end of the recommendation list. Currently, CNN has a good effect on minit datasets, but there are still some bugs. Because it is busy recently, I will summarize the previous issues and continue to optimize them later.

Convolutional Neural Network (CNN) is an important algorithm for deep learning. It has outstanding results in many applications. [1] It compares the effect of multiple algorithms in character recognition of documents, the conclusion is that CNN is better than all other algorithms. CNN achieves the best results in handwriting recognition. [2] CNN is applied to face-based gender recognition, and the results are also very good. Some time ago, I used a back-propagation neural network to identify the numbers in the pictures taken by my mobile phone. The effect was pretty good, close to 98%, but it did not do well in Chinese Character Recognition. So I wanted to try a convolutional neural network.

1. The overall network structure of CNN

The convolutional neural network is an improvement in the back-propagation neural network. Similar to the back-propagation neural network, both use Forward propagation to calculate output values, and reverse propagation to adjust weights and offsets; the biggest difference between CNN and standard BP is that the neural units in the adjacent layers of CNN are not fully connected, but partially connected, that is, the sensing area of a neural unit comes from some neural units in the upper layer, rather than being connected to all neural units as BP. CNN has three important ideological architectures:

• Local Region awareness
• Weight sharing
• Spatial or temporal sampling

Local Area sensing can discover some local features of data, such as an angle on the chip and an arc. These basic features constitute the basis of animal vision [3]. In bp, all pixels are chaotic points, and the relationships between them are not mined.

Each layer of CNN is composed of multiple maps, and each map is composed of multiple neural units. All the neural units of the same map share a convolution kernel (that is, the weight ), convolution kernels often represent a feature. For example, if a convolution and a section of an arc are involved, roll the convolution kernel over the entire image and the area with a large convolution value may be a section of an arc. Note that the convolution kernel is actually a weight. We do not need to calculate a convolution separately, but a fixed-size weight matrix to match images. This operation is similar to convolution, therefore, we call it a convolutional neural network. In fact, BP can also be seen as a special convolutional neural network. However, this convolutional core is the ownership weight of a layer, that is, the sensing area is the entire image. The weight sharing policy reduces the parameters to be trained, making the trained model more general.

The main purpose of sampling is to confuse the specific location of a feature, because the specific location of a feature is no longer important after it is found. We only need the relative location of this feature and other features, for example, when we get the above "O", we don't need to know its location in the image, we only need to know that there is an "O" under it, so we can know it is an '8', because the picture "8" in the picture is left or right does not affect our understanding of it, this obfuscation policy can identify distorted and distorted images.

CNN has strong robustness to the distortion of input data in space (mainly for image data) and time (mainly for time series data, refer to tdnn. CNN generally uses the convolution layer and the sampling layer to alternate settings, that is, a convolution layer is connected to a sampling layer, and the sampling layer is followed by a convolution layer... in this way, the convolution layer extracts features, combines them to form more abstract features, and finally forms the descriptive features of the image object. CNN can also be followed by the full connection layer, the full connection layer is the same as BP. The following is an example of a convolutional Neural Network:

Figure 1 (image source)

This is the basic idea of Convolutional neural networks, but there are multiple implementations. I have referred to the deep learning toolbox deeplearntoolbox of Matlab, the biggest difference between CNN and Other implementations is that the sampling layer has no weight or bias, and only performs a sampling process on the convolution layer. The test dataset of this toolbox is minist, each image is 28*28 in size, which implements the following CNN:

Figure 2

2. network initialization

CNN initialization mainly initializes the convolution kernel (weight) and bias of the convolution layer and the output layer. In deeplearntoolbox, the convolution kernel and weight are randomly initialized, and the bias is fully initialized by 0.

3. forward transmission computing

The input layer, convolution layer, sampling layer, and output layer are calculated in different ways during forward computation.

　　3.1 input layer: The input layer has no input value and only one output vector. The vector size is the image size, that is, a 28*28 matrix;

　　3.2 convolution Layer: The convolution layer input is either from the input layer or from the sampling layer, such as the red part. Each map in the convolution layer has a convolution kernel of the same size. The Toolbox contains 5x5 convolution kernels. The following is an example. For the sake of simplicity, the convolution kernel size is 2*2, and the feature map size of the previous layer is 4*4, which is used to roll over the image, obtain a map of features (4-2 + 1) * (4-2 + 1) = 3*3. the convolution kernel moves one step at a time. In the Toolbox implementation, a map in the convolution layer is associated with all the maps in the upper layer, such as S2 and C3, that is, C3 has 6*12 convolution cores, each feature map in the convolution layer is a convolution of different convolution kernels on all the maps in the previous layer, and adds an offset after accumulating the corresponding elements, and then obtains sigmod. Note that the number of maps in the convolution layer is specified during network initialization, and the size of the map in the convolution layer is determined by the size of the convolution core and the input map of the previous layer, assume that the map size of the previous layer is N * n and that of the convolution kernel is K * K, then the map size of the layer is (n-k + 1) * (n-k + 1). For example, the map size of 24*24 is 24 = (28-5 + 1 ). Stanford's deep learning tutorial details the computation process of convolution Feature Extraction in more detail.

Figure 3

　　3.3 subsampling): The sampling layer is a sampling process for the previous map. The sampling method here is to aggregate statistics on the adjacent small areas of the previous map. The area size is scale * scale, some implementations take the maximum value of a small area, while the implementation in toolbox uses the average value of 2*2 small areas. Note that the convolution calculation windows overlap, while the calculation windows do not overlap. The calculation samples in the toolbox also use convolution (conv2 (A, K, 'valid ')) to achieve this, the convolution kernel is 2*2, and each element is 1/4. Remove the overlapping part of the calculated convolution result, that is:

Figure 4

4. Adjust the weight of reverse Transmission

The reverse transmission process is the most complex part of CNN. Although the basic idea is the same as that of BP, the weights and offsets are adjusted by minimizing the residual, however, the network structure of CNN is not as simple as that of BP, and the processing methods for different structures are different. In addition, the weight sharing makes computing residual much more difficult, many papers [1] [5] and [4] have detailed descriptions, but I found that some details are still not clear, especially the residual computation on the sampling layer, I will describe it in detail here.

4.1 residual of the output layer

Like BP, the residual calculation method of the output layer of CNN is different from that of the intermediate layer. the residual value of the output layer is the error value of the output value and the class mark value, the residual values of the intermediate layers are derived from the weighted sum of the residual values of the next layer. The residual calculation of the output layer is as follows:

Formula Source

This formula is not explained. You can view the formula source and the interpretation of Stanford's deep learning tutorial.

4.2 residual of the convolution layer of the next layer as the sampling layer (subsampling)

When the next layer (L + 1) of a convolutional layer L is the sampling layer, assuming that we have calculated the residual of the sampling layer, we can calculate the residual of the convolutional layer. From the top network structure, we know that the map size of the sampling layer (L + 1) is 1/(scale * scale) of the convolution layer L. in the toolbox, scale 2, however, the number of maps in the two layers is the same. The four units in a map in the convolution layer L are associated with one unit in the map corresponding to the L + 1 layer, the residual data in the sampling layer can be expanded to a scale * scale all-1 matrix, so that the residual dimension of the sampling layer is consistent with that of the output map on the previous layer, the Toolbox code is as follows, where D represents the residual, and a represents the output value:

net.layers{l}.d{j} = net.layers{l}.a{j} .* (1 - net.layers{l}.a{j}) .* expand(net.layers{l + 1}.d{j}, [net.layers{l + 1}.scale net.layers{l + 1}.scale 1])

Scaling process:

Figure 5

Use convolution to calculate the residue of the convolution layer:

Figure 6

4.3 The next layer isConvolutionLayer (subsampling)SamplingLayer residual

　　When the next layer of a sample layer L is a convolutional layer (L + 1), assuming that we have calculated the residual of the L + 1 layer, now we calculate the residual of the L layer. The direct connection from the sampling layer to the convolution layer is weighted and biased, so it is not as simple as connecting from the convolution layer to the sampling layer. Now let's assume that the J map MJ in layer L is associated with m2j in layer L + 1. According to the BP principle, the residual DJ in layer L is the weighted sum of residual d2j in layer L + 1, however, the difficulty here is that it is difficult for us to clarify the weights of those m2j units associated with those MJ units, and convolution (slightly deformation) is used in the toolbox) the code for cleverly solving this problem is:

convn(net.layers{l + 1}.d{j}, rot180(net.layers{l + 1}.k{i}{j}), ‘full‘);

Rot180 indicates that the matrix is rotated 180 degrees (through symmetric row switching and column symmetric switching). Why do we need to rotate the convolution kernel here? The answer is: Through this rotation, in the 'full' mode, convolution exactly captures the relationship between the upper-layer map unit of forward transmission computing and convolution and the current layer map, note that convn, the built-in function of Matlab, performs a rotation on the convolution kernel before computation of convolution. Therefore, all the previous convolution calculations rotate the convolution kernel:

a =     1     1     1     1     1     1     1     1     1k =     1     2     3     4     5     6     7     8     9>> convn(a,k,‘full‘)ans =     1     3     6     5     3     5    12    21    16     9    12    27    45    33    18    11    24    39    28    15     7    15    24    17     9

Convn also expands the convolution matrix by 0 before calculation. If the convolution kernel is K * K, the convolution matrix is N * n, need to be expanded to (n + 2 (k-1) * (N + 2 (k-1) with N * n + 2 () as the center, all above convn (A, K, 'full') is calculated as follows:

Figure 7

In fact, whether or not the convn internal rotation has no impact on network training, as long as the internal consistency (that is, both rotate or do not rotate), all of my convolution implementations do not rotate the convolution kernel. If convn is first rotated to 180 degrees before calculation, and then convn is rotated to 180 degrees internally, it is equivalent to not changing the convolution kernel.

To clearly describe the weights and units associated with convolution of convolution kernel rotation 180 and convolution layer residual, It is the weights and units associated with forward calculation, we select a slightly larger convolution kernel, that is, assume that the convolution layer uses a 3*3 convolution kernel, and the output map of the previous sampling layer is 5*5, the process of obtaining the convolution layer from the sampling layer for forward transmission is as follows:

Figure 8

Here we use the self-implemented convn (that is, the convolution kernel is not rotated internally), and assume that the bottom labels of the matrix A and B above start from 1, then there are:

B11 = A11*K11 + A12*K12 + A13*K13 + A21*K21 + A22*K22 + A23*K23 + A31*K31 + A32*K32 + A33*K33B12 = A12*K11 + A13*K12 + A14*K13 + A22*K21 + A23*K22 + A24*K23 + A32*K31 + A33*K32 + A34*K33B13 = A13*K11 + A14*K12 + A15*K13 + A23*K21 + A24*K22 + A25*K23 + A33*K31 + A34*K32 + A35*K33B21 = A21*K11 + A22*K12 + A23*K13 + A31*K21 + A32*K22 + A33*K23 + A41*K31 + A42*K32 + A43*K33B22 = A22*K11 + A23*K12 + A24*K13 + A32*K21 + A33*K22 + A34*K23 + A42*K31 + A43*K32 + A44*K33B23 = A23*K11 + A24*K12 + A25*K13 + A33*K21 + A34*K22 + A35*K23 + A43*K31 + A44*K32 + A45*K33B31 = A31*K11 + A32*K12 + A33*K13 + A41*K21 + A42*K22 + A43*K23 + A51*K31 + A52*K32 + A53*K33B32 = A32*K11 + A33*K12 + A34*K13 + A42*K21 + A43*K22 + A44*K23 + A52*K31 + A53*K32 + A54*K33B33 = A33*K11 + A34*K12 + A35*K13 + A43*K21 + A44*K22 + A45*K23 + A53*K31 + A54*K32 + A55*K33

We can obtain the associations between each unit of matrix B and which convolution kernel units and which a matrix units are:

A11 [K11] [B11]A12 [K12, K11] [B12, B11]A13 [K13, K12, K11] [B12, B13, B11]A14 [K13, K12] [B12, B13]A15 [K13] [B13]A21 [K21, K11] [B21, B11]A22 [K22, K21, K12, K11] [B12, B22, B21, B11]A23 [K23, K22, K21, K13, K12, K11] [B23, B22, B21, B12, B13, B11]A24 [K23, K22, K13, K12] [B23, B12, B13, B22]A25 [K23, K13] [B23, B13]A31 [K31, K21, K11] [B31, B21, B11]A32 [K32, K31, K22, K21, K12, K11] [B31, B32, B22, B12, B21, B11]A33 [K33, K32, K31, K23, K22, K21, K13, K12, K11] [B23, B22, B21, B31, B12, B13, B11, B33, B32]A34 [K33, K32, K23, K22, K13, K12] [B23, B22, B32, B33, B12, B13]A35 [K33, K23, K13] [B23, B13, B33]A41 [K31, K21] [B31, B21]A42 [K32, K31, K22, K21] [B32, B22, B21, B31]A43 [K33, K32, K31, K23, K22, K21] [B31, B23, B22, B32, B33, B21]A44 [K33, K32, K23, K22] [B23, B22, B32, B33]A45 [K33, K23] [B23, B33]A51 [K31] [B31]A52 [K32, K31] [B31, B32]A53 [K33, K32, K31] [B31, B32, B33]A54 [K33, K32] [B32, B33]A55 [K33] [B33]

Then, use the convn of MATLAB (the convn kernel is rotated 180 degrees internally) to perform a convn (B, K, 'full'). In combination with figure 7, check the red part, remove 0, a11 = B '33 * K' 33 = B11 * K11. It is found that A11 is related to K11 and B11, right; let's look at another A24 = B '34 * k'21 + B' 35 * k'22 + B '44 * k'31 + B' 45 * k'32 = B12 * k23 + B13 * k22. + B22 * k13 + B23 * K12, we found that the convolution kernel units involved in the A24 calculation and the B Matrix units are exactly the units associated with the forward computation. Therefore, we can obtain the residue of the sampling layer by rotating the convolution kernel and then performing convolution.

After the residual computation, the rest is to update the weight and offset, which is the same as BP, so we will not go into details. If you have any questions, please contact us.

5. Code Implementation

The detailed code is no longer posted here. I am still on GitHub. Please refer to and correct me. I have re-built the wheel again and have not used any third-party library class. Here I will post the call code:

Public static void runcnn () {// create a convolutional Neural Network layerbuilder builder = new layerbuilder (); builder. addlayer (layer. buildinputlayer (new size (28, 28); builder. addlayer (layer. buildconvlayer (6, new size (5, 5); builder. addlayer (layer. buildsamplayer (new size (2, 2); builder. addlayer (layer. buildconvlayer (12, new size (5, 5); builder. addlayer (layer. buildsamplayer (new size (2, 2); builder. addlayer (layer. buildoutputlayer (10); CNN = new CNN (builder, 50); // import the dataset string filename = "dataset/train. format "; dataset = dataset. load (filename, ",", 784); CNN. train (dataset, 3); // string modelname = "model/model. CNN "; CNN. savemodel (modelname); dataset. clear (); dataset = NULL; // prediction // CNN = CNN. loadmodel (modelname); dataset testset = dataset. load ("dataset/test. format ",-1); CNN. predict (testset, "dataset/test. predict ");}

6. References

[1]. Yann lecun. Gradient-based learning applied to document recognition.

[2]. Shan upgrade Liew. gender classification: A convolutional neural network approach.

[3] D. H. Hubel and T. N. Wiesel, "valid tive fields, binocular interaction teraction, and functional architecture in the cat's visual cortex ,"

Tornadomeet. http://www.cnblogs.com/tornadomeet/p/3468450.html.

[5] Jake bouvrie. Notes on convolutional neural networks.

[6] the detailed introduction of C ++ implementation. http://www.codeproject.com/Articles/16650/Neural-Network-for-Recognition-of-Handwritten-Digi

[7] MATLAB deeplearntoolbox https://github.com/rasmusbergpalm/DeepLearnToolbox

 

Reprinted please indicate the source: http://www.cnblogs.com/fengfenggirl