The development, introduction, Contribution of neural network-googlenet

The development of Googlenet inception V1:

The well-designed Inception Module in the Inception V1 improves the utilization of the parameters, Nception V1 removes the final fully connected layer of the model, using the global average pooling layer (which changes the image size to 1x1), in the previous network, The whole connection layer occupies most of the network parameters, it is easy to produce the phenomenon of fitting; (see below for a detailed analysis)

Inception V2:

Inception V2 studied vggnet, replaced the large convolution nucleus of 5*5 with two 3*3 convolution (reducing the number of parameters while reducing the cross-fitting), and also proposed the Batch normalization(abbreviation bn) method. BN is a very effective regularization method, which can accelerate the training speed of large convolutional networks many times, and the classification accuracy of convergence can be greatly improved.

When used in a layer of a neural network, BN standardizes the interior of each mini-batch data to normalize the output to a normal distribution (0,1), reducing the internal covariate shift (changes in the distribution of neurons in the interior). bn thesis points out that the traditional deep neural network training, each layer of the input distribution is changing, resulting in training is difficult, we can only use a very small learning rate to solve the problem. And for each layer using bn, we can effectively solve this problem, the learning rate can be increased many times, the accuracy between the need to achieve the number of iterations need 1/14, the training time is greatly shortened, and after reaching the accuracy rate, you can continue training. Think bn in a sense also played a regularization role, all can reduce or cancel dropout, simplify the network structure.

Of course, when using bn, some adjustments are needed :

• Increase learning rate and accelerate learning attenuation to fit bn normalized data
• Removal of dropout and reduction of L2 regular (Bn has played a regularization role)
• Removing LRN
• Shuffle the training samples more thoroughly
• Reduce optical distortion of data during data enhancement (BN training is faster and each sample is trained less often, so a real sample is more helpful for training)

Inception V3:

Inception V3 is mainly reformed in two areas:

• The idea of factorization into small convolutions was introduced to split a larger two-dimensional convolution into two smaller one-bit convolution, such as splitting the 7*7 convolution into 1*7 convolution and 7*1 convolution (3*3 split into 1*3 and 3*1). on the one hand, a large number of parameters are saved, the acceleration operation is subtracted from the fitting, and the expression ability of a nonlinear extended model is added . It is pointed out in the paper that the result of asymmetric convolution structure is more obvious than that of symmetric splitting into several identical small convolution nuclei, which can deal with more and richer spatial features and increase the feature diversity.

3*3 convolution cores are split into 1*3 convolution and 3*1 convolution:

• On the other hand, Inception V3 optimizes the structure of the Inception module, and now Inception module has three different structures such as 35*35, 17*17, and 8*8. These inception module only appear at the back of the network, and the front is a normal convolution layer. Branches are also used in the branches of the inception module.

Inception Module for three structures in Inception V3:

Inception V4:

Inception V4 compared to V3 is mainly a combination of Microsoft's ResNet, interested can view "inception-v4, Inception-resnet and the Impact of residual Connections on Learning "paper.

The development, introduction, Contribution of neural network-googlenet