"CV paper reading" Going Deeper with convolutions (googlenet)

Objective:

Improve the performance of deep neural networks.

The general approach poses problems:

Increase the depth and width of the network.

Brings two questions:

(1) Parameter increase, the situation of insufficient data can easily lead to overfitting

(2) High computational resource requirements, and in the training process will make many parameters tend to 0, waste computing resources.

Workaround:

Use sparse joins instead of dense structures.

theoretical basis (ARORA): a probability distribution can be represented by a large sparse depth neural network, and the optimal structure is constructed by analyzing the statistical correlations of the activation state of the upper layer and aggregating the output height-dependent neurons. This is consistent with the Hebbian rule in biology, "some neurons respond in a consistent, simultaneous excitement or inhibition."

There is a problem: The computer's infrastructure is inefficient when it encounters sparse data calculations, and the use of sparse matrices makes it much less efficient.

Objective: to design a network structure that can exploit both sparsity and dense computing.

Inception Model:

How is sparsity used in the model? I am not sure, but it is estimated that the different cores are used at the same level to carry out convolution of the input feature, dispersed into several small tasks, and then converge. Such as:

This is the most primitive model. As can be seen, because the convolution does not necessarily change the size, and the number of channels due to the decentralized connection will eventually increase, it is easy to cause the number of parameters of the exponential level rise. In this paper, the use of the NIN network mentioned in the use of 1*1 convolution nucleus reduction of the role of the convolution layer before processing, the feature layer is reduced dimension (note is the channel dimensionality reduction, not the dimensionality of the space), such as the original M channel, descending dimension to the P channel, in the convergence into the M channel, At this point the number of parameters does not grow exponentially with depth, such as:

The rationality of this is that the Hebbin rule says "some neurons are excited or suppressed at the same time", while in the region the corresponding region of the same node may be the same, think that they are related, so through the 1*1 convolution nucleus will be aggregated (information compression) and then convolution, in line with Arora theory. Also, notice that there is a maximized pooling layer.

The benefits of this processing are (1) depth increase, the number of nodes controllable (2) There are multiple scales such as 3*3,1*1,5*5,7*7.

Googlenet structure:

The Googlenet network has 22 layers, and the last layer uses the global average pooling layer in the NIN network, but the FC layer is added and then entered into the Softmax function. Such as:

In the case of deep deepening, when the BP algorithm executes, some gradients may be 0, which slows down the convergence of the network. The method used in this paper is to add two output layers (auxiliary classifiers) so that the gradient of some weights will come from the superposition of multiple parts and accelerate the convergence of the network. But when you predict it, the AC layer is removed.

"CV paper reading" Going Deeper with convolutions (googlenet)