Convolutional Neural Networks (ii)

Originating from: http://neuralnetworksanddeeplearning.com/.

convolutional neural networks employ three basic concepts: local receptive fields, shared weights (sharedweights), and blending (pooling).

Local sensation Field: In the previously seen network of fully connected layers, the input is depicted as a vertically arranged neuron. But in a convolutional network, it is more helpful to think of input as a 28x28 square-arranged neuron whose value corresponds to the pixel light intensity of the 28x28 we use as input:

As usual, we connect the input pixels to a hidden neuron layer. But we don't connect each input pixel to each hidden neuron. Instead, we simply connect the input image to a small, local area. To be exact, each neuron in the first hidden layer is connected to a small area of an input neuron, for example, a 5x5 area that corresponds to 25 input pixels. So for a particular hidden neuron, we might have a connection that looks like this

The area of the input image is known as the local sensation field of the hidden neurons. It is a small window on the input pixel. Each connection learns a weight. The hidden neurons also learn a general bias. You can think of this particular hidden neuron as learning to analyze its local sensations.

We then cross-move the local sensation field across the entire input image. For each local sensation field, there is a different hidden neuron in the first hidden layer. For the right explanation, let's start with a local sensation from the top left:

We then move the local sensation field to the right pixel (that is, a neuron) and connect to the second hidden neuron:

So repeat, build up the first hidden layer. Note that if we have a 28x28 input image and a 5x5 local sensation field, then there will be 24x24 neurons in the hidden layer. This is because before reaching the right (or bottom) of the input image, we can only move the local sensation field laterally 23 neurons (or 23 neurons down). I'm showing a local sensation field that moves one pixel at a time. In fact, sometimes different spans are used. For example, I can move to the right (or down) 2 pixels of the local sensation field, in which case we used 2 spans. Most of the time in this chapter we will use a 1 span, but it is worth knowing that people sometimes use different span tests.

Share weights and offsets: I've said that each hidden neuron has a bias and a 5x5 weight attached to its local feel field. What I did not mention is that we intend to use the same weights and biases for each of the 24x24 hidden neurons.

In other words, for the J, K hidden neurons, the output is:

Here σ is the activation function of neurons--it can be the S-type function we used in the previous chapter. B is the shared value of the bias. Wl,m is a 5x5 array of shared weights. Finally, we use ax,y to represent the input activation value for position x, Y.

This means that all neurons in the first hidden layer detect exactly the same characteristics, just at different points in the input image. To understand why this is the reason, think of weights and biases as hidden neurons can pick things, for example, in a specific local sense of the vertical edge of the wild. This ability is also likely to be useful in other locations of the image. Therefore, it is very useful to apply the same feature detector in the image. With a slightly more abstract term, convolutional networks can adapt well to the translation invariance of images: for example, by moving a cat image slightly, it is still a picture of a cat.

For this reason, we sometimes refer to mappings from the input layer to the hidden layer as a feature map. We refer to the weighting of defining feature mappings as shared weights. The bias that defines a feature map in this way is called a shared bias. Shared weights and biases are often referred to as a convolution core or filter. In the literature, people sometimes use these terms in a slightly different way, and I'm not going to make a strict distinction; we'll look at some concrete examples later.

The network structure I describe at the moment can only detect a type of local feature. In order to complete the image recognition we need more than one feature map. So a complete convolution layer consists of several different feature mappings:

In this example, there are 3 feature mappings. Each feature map is defined as a set of 5x5 shared weights and a single shared bias. The result is that the network can detect 3 different features, each of which can be detected throughout the image.
To make the above diagram simple, I only showed 3 feature mappings. In practice, however, convolutional networks may use many (perhaps much more) feature mappings. An early identification MNIST digital convolutional network, LENET-5, uses 6 feature mappings, each associated to a 5x5 local sensation field. So the illustration above is actually very close to the LeNet-5. In the example we'll develop later in this chapter, we'll use a convolution layer with 20 and 40 feature mappings. Let's take a quick look at some of the features we've learned:

These 20 images correspond to 20 different feature mappings (or filters, cores). Each map has an image representation of a 5x5 block, corresponding to the 5x5 weight of the local sensing Nonaka. The white block means a small (typical, smaller negative) weight, so the feature map has a smaller response to the corresponding input pixel. A darker block means a larger weight, so the feature map has a greater response to the corresponding input pixel. Very roughly speaking, the above image shows the type of feature that the volume base responds to.

So what conclusions can we get from these feature mappings? It is clear that there is a space structure beyond our expectation: Many of these features have clear bright and dark subregions. This means that our network is actually learning something related to spatial structure. However, apart from that, it is difficult to see what these feature detectors are learning. Of course, we are not learning (for example) Gabor filters, which have been used in many traditional methods of image recognition. In fact, there are a lot of work on the convolution network to better understand the characteristics of the product. If you are interested, I suggest starting with Matthew Zeiler and Rob Fergus's (2013) thesis visualizing and understanding Convolutional Networks.

One of the great advantages of sharing weights and biases is that it greatly reduces the parameters of participating convolutional networks. For each feature map we need to include a 5x5 share weight, plus a shared bias. So each feature map requires 26 parameters. If we have 20 feature mappings, then there is a total of 20x26 = 520 parameters to define the convolution layer. In contrast, suppose we have a fully connected first layer with 784 = 28x28 input neurons, and a relatively modest 30 hidden neurons, as we used in many examples before this book. There are a total of 784x30 weights, plus an additional 30 offsets, a total of 23, 550 parameters. In other words, this fully connected layer has up to 40 times times the parameters of the volume base.

Of course, we can't really do a direct comparison between the number of parameters, because the nature of the two models is different. However, intuitively, panning invariance using convolutional layers seems likely to reduce the number of parameters that achieve the same performance in the fully connected model. In turn, this will lead to faster convolutional model training and, ultimately, will help us to build deep networks using convolutional layers.

Mixed layer: In addition to the convolution layer just described, convolutional neural networks also contain a mixed layer (pooling layers). The mixing layer is usually immediately followed by the convolution layer. What it does is to simplify the information that is output from the convolution layer.

In detail, a mixed layer obtains each feature map from the convolution layer output, and prepares a condensed feature map from them. For example, each unit of a blend layer may summarize one (for example) 2x2 area of the previous layer. As a concrete example, a frequent mixing program is called the Maximum Mix (max-pooling). In the maximum blending, a mixing unit simply outputs the maximum activation value of its 2x2 input area, as stated:

Note that since the convolution layer has a 24x24 neuron output, we get 12x12 neurons after mixing.

As mentioned above, the convolution layer usually contains more than one feature map. We will mix the maximum values for each feature map, respectively. So if there are three feature mappings, the combined convolution layer and the maximum value

The blending layer looks like this.

We can think of the maximum mix as a way for a network to ask if there is a given feature in the area of an image where it is found. Then it throws out the exact location information. Intuitively, once a feature is found, its exact position is not as important as its approximate position with respect to other features. A big benefit is that this can have many features that are less mixed, so this helps to reduce the number of parameters required at a later layer.

Maximum blending is not the only technique used for blending. Another common method is L2 blending (L2 pooling). Here we take the square root of the square of the active value in the 2x2 area, not the maximum activation value. Although the details are different, their intuitive and maximum blending is similar: L2 blending is a way to shrink the information that is exported from the convolution layer. In practice, both technologies are widely used. And sometimes people use other types of mixed operations. If you are trying to optimize performance, you can use validation data to compare different methods of blending and choose one that works best. But we are not going to miss this detail on optimization.

Together: We can now put these ideas together to construct a complete convolutional neural network. It's similar to the architecture we just saw, but there is an extra layer of 10 output neurons, corresponding to 10 possible MNIST numbers (' 0 ', ' 1 ', ' 2 ', etc.):

The network starts with 28x28 input neurons, which encode the pixel strength of the MNIST image. Next is a convolution layer, using a 5x5 local sensing field and 3 feature mappings. The result is a 3x24x24 hidden feature neuron layer. The next step is a maximum blending layer, applied to the 2x2 region, spanning 3 feature mappings. Results
is a 3x12x12 hidden feature neuron layer.

The last connected layer in the network is an all-connected layer. Rather, this layer connects each neuron of the maximum mixing layer to each output neuron. This fully-connected structure is the same as that used in our previous chapters. However, note the above illustration, in order to simplify, I only use an arrow instead of displaying all the connections. Of course, you can easily imagine these connections.

This convolution schema is quite different from the schema used in previous chapters. But the overall description is similar: A network of many simple units whose behavior is determined by their weights and biases. The overall goal is still the same: training data to train the weight and bias of the network, so that the network can be qualified to enter the number of categories.

In particular, as we did earlier in this book, we will train our network with a gradient descent and reverse propagation. This is mostly handled in exactly the same way as in the previous chapters. However, we do need to make some changes to the reverse-propagation program. The reason is that our previous derivation of the reverse propagation is for a fully connected layer of the network. Fortunately, the derivation of the convolution and maximum blending layers is straightforward. If you want to understand the details, then I ask you to complete the following questions. Note that this problem will take some time to complete, unless you really have absorbed the derivation of the previous reverse propagation (which is easy in this case).

Problem
• Reverse propagation in convolutional networks in a network with a fully connected layer, the core equation for reverse propagation is (BP1) – (BP4) (link). Suppose we have a network that contains a convolution layer, a maximum blending layer, and an all-connected output layer, as discussed above. How to modify the inverse propagation equation?

Convolutional Neural Networks (ii)