convolutional neural Networks (2): Sparse Interactions, receptive Field and Parameter sharing

Sparse Interactions, receptive Field and Parameter sharing are the core parts of the entire CNN depth network, and we use this article to specifically analyze its principles.

First we consider the output matrix of the Feedforward neural network,l layer, which is equal to the input matrix of L layer and the weight matrix of L layer to do matrix multiplication, then the nonlinear transformation. In other words, each output data of the L layer is related to each input data of the L layer. If the input data is m-dimensional and the output data is n-dimensional, there are m*n weights to characterize the relationship between input and output. Therefore, the time complexity of Forward-propagation is O (m*n).

According to the actual experience in machine learning, weight too many problems mainly include: difficult to train, overfitting and so on. So CNN introduced sparse interactions to solve the problem of dense weights (dense Weight). Take an example of a network with a core width of 3: The next layer of output S3, only with 3 input x2,x3,x4.

X2,x3,x4 called S3 's receptive Field, which is actually an exotic concept from neuroscience. Here's a quote from Wikipedia about receptive field:

Hubel and Wiesel in the 1950s and 1960s showed, Cat and monkey visual cortexes contain neurons that Individua Lly respond to small regions of the visual field. Provided the eyes is not moving, the region of visual space within which visual stimuli affect the firing of a single neu Ron is known as its receptive field. Neighboring cells has similar and overlapping receptive fields. Receptive field size and location varies systematically across the cortex to form a complete map of visual space. The cortex in each hemisphere represents the contralateral visual field. "

In the cerebral visual cortex (visual Cortex), which is located in the occipital lobe (occiptal Lobe), there are neurons, each of which corresponds to a receptive Field. This is where biological neural network inspires artificial neural network in image processing. If we just end up, we do reduce the complexity of the model and reduce the weight matrix to k*n,k is the size of kernel. In other words, there are n neurons in the next layer, each neuron only related to the K values of the previous layer.

However, the introduction of the concept of weight sharing, the model is further simplified to achieve: the number of weight is only related to the size of kernel. For kernel and Weight sharing, it can be understood that there is no fixed connection between the L layer and the L-1 layer, but rather dynamic binding, where there is a small window between the two layers, called kernel. A small portion of the original image can be seen through the window, and the entire picture is scanned as the window continues to slide from left to right, top to bottom. While scanning to the picture area, and kernel to do convolution, generate feature map. The whole picture, the shared weight is actually the value of kernel, if the kernel change, the entire map of the scan results (feature map) will change. Process such as:

In this step, I personally think that can solve the first w that what the problem, but about why, how, the follow-up will require a lot of space and time, slowly to learn and analyze. This is the second of CNN, not to be continued.

convolutional neural Networks (2): Sparse Interactions, receptive Field and Parameter sharing