V-Bomb: Vgg

Next to a model that is often used today, from Oxford University's Vgg, the network currently has a lot of improved versions, and here are just a few of the original models, from the thesis analysis and model understanding.

I. Analysis of the thesis

A: summary

......

The network developed from alex-net mainly modifies two aspects:

1, use smaller filter size and spacing in the first volume base layer;

2, train and test the picture on the whole picture and multi-scale.

Two: Network configuration

2.1 Configuration

2.1.1 Small filter size is 3*3

The volume base of the convolution interval s=1;3*3 has a 1-pixel padding.

1:3*3 is the smallest size capable of capturing the upper and lower left and center concepts.

2: Two 3*3 of the volume base of the limited feeling field is 5*5, three 3*3 is 7*7, can replace the large filter size

3: Multiple 3*3 volume base has more nonlinearity than a large size filter roll base, which makes the decision function more decision-making.

4: More than one 3*3 convolution layer than a large size filter has less parameters, assuming that the input and output of the volume base of the characteristics of the same size as C, then three 3*3 of the convolution layer parameters (3*3*c*c) =27cc; a 7*7 convolution layer parameter is 49CC; so you can put three 3 The Filter of * * is considered as a decomposition of a 7*7filter (the middle layer has a nonlinear decomposition).

2.1.2 1*1 Filter:

The function is to linearly deform the input line without affecting the number of input and output dimensions, then the nonlinear processing by Relu to increase the nonlinear expression ability of the network.

Pooling:2*2, Interval s=2;

2.2 Structure

And the previous popular three-stage network is not, this article is a 5 max-pooling layer, so is the 5-stage convolution feature extraction. The number of convolution per layer starts at 64 of the first stage, and each stage grows by one time, until the highest 512 is reached and then maintained.

Basic structure A:

Input (224,224,3) →64f (3,3,3,1) →max-p (2,2) →128f (3,3,64,1) →max-p (2,2) →256f (3,3,128,1) →256f (3,3,256,1) →max-p (2,2) →512f (3,3,256,1) →512f (3,3,512,1) →max-p (2,2) →512f (3,3,256,1) →512f (3,3,512,1) →max-p (2,2) →4096fc→4096fc→1000softmax

8 volume base, 3 full-connection layer, total 11 layers; The author only explains the reasons for using 3*3filter, as to the number of layers, stages, and characteristics of why the features are designed, the author does not specify.

Number of parameters: Network E and overfeat model parameters almost

B: Add a 3*3 volume base in Stage2 and Stage3 of a, 10 convolutional layers, total 13 layers

C: On the basis of B, stage3,stage4,stage5 respectively increased the 1*1 convolution layer, 13 volume base, total 16 layers

D: On the basis of C, stage3,stage4,stage5 respectively increased the convolution layer of 3*3, 13 volume base, a total of 16 layers

E: On the basis of D, stage3,stage4,stage5 respectively increased the convolution layer of 3*3, 16 volume base, a total of 19 layers

Third, classification framework

3.1 Training parameter settings

MINIBATCH=256, the others are the same.

The authors found that although Vgg has more parameters and deeper levels than alex-net, Vgg requires very few iterations to begin to converge. This is because

1, depth and small filter size play an implicit role in regulation

2, some layers of pre-initialisation

Pre-initialisation: The Gaussian distribution of the weights w~ (0,0.01) of network A, bias is 0, and because of the large number of Relu functions, the initial value of bad weights has great influence on the network training. To circumvent this problem, the author now trains the lightest network A in a random way, and then, while training other networks, takes the first 4 volumes of a (the sense that each stage takes the first convolution) and the last fully connected layer as the initial value of the other network, and the unassigned middle tier is randomly initialized.

Multi-scale Training

Zoom the original image to the minimum edge s>224; and then extract the 224*224 fragment on the full image for training.

Method 1: Train two models on s=256, and s=384, and then ask for an average

Method 2: Similar to the method used in Overfeat testing, on [Smin,smax]scale, randomly select a scale, and then extract 224*224 pictures, training a network. This method is similar to the data gain on the image size.

3.2 Testing

The test phase method and the Overfeat test method are the same, first select a scale:q, and then apply convolutional networks on this image, generate unpooled FM in the last convolution phase, and then use the sliding window method, each pooling Window produces a categorical output and then merges the results of each pooling window to get the final classification. This is more efficient than 10-view, and it is only a one-time convolution process.

3.3 Details of the deployment

Using C + + Caffe toolbox, parallel Computing on 4 Titan GPUs is 3.75 times times faster than a single GPU, and almost 2-3 weeks per network.

Four, the classification experiment

4.1 Test Phase Single-scale Comparison

A vs A-LRN:A-LRN results without A good, which means that LRN does not work very well.

A vs B,c,d,e: The Deeper the better

A vs C: Increase the 1*1filter, that is, increase the additional non-linearity does enhance the effect

The C vs d:3*3 filter is better than 1*1filter, and using larger filter captures larger spatial features.

Training method: The scale range [256;512] is used to train the network through the gain of the ratio, which results in a significant improvement over the fixed two s=256 and s=512. Multi-scale training is really useful, because in ZF's thesis, convolutional networks have a certain invariance in scaling, and the ability to do so can be increased by multi-scale training.

4.2 Multi-scale Training

Method 1:single-scale Training S,multi-scale test {s-32,s,s+32}

Method 2:multi-scale Training [Smin; Smax],multi-scale Test {Smin,middle,smax}

Result: The result here is B '

1 B vs B ', C vs C ',......: Single-scale training, using Multi-scale test, has a 0.2% top-5 boost.

2 B-256 vs B-384 ... : Single-scale training on 256 and 348, no matter what test method, the results are almost the same. This indicates that the network has limited extracting capacity on a single scale.

3 Multi-scale training, multi-scale test, for the network promotion is obvious, D ' and E ' top-5 classification reached 7.5%.

4.3 Model fusion

By averaging the results, the results of the above different networks are fused.

The result of Model fusion is, more interestingly, the result of fusion of model D and e two top models is better than the result of merging 7 models. This is more interesting, model fusion number, but no two network good. Why is this? didn't want to understand.

4.4 Comparison with other networks

There are some gaps between the results of this article and the results in post 9, which may be the reason for training platforms and methods, and different training platforms and methods have an impact on the results.

Five, positioning

5.1 Locating the network

Similar to the overfeat approach, using Model D (least of the parameters) replaces the classifier with a regression function, two classification methods: SCR (Single-classregression), and a regression function to learn to predict all categories of bounding box PCR (per-class regression) Each category has its own separate regression function.

Training: Training two models on s=256 and s=384 respectively, the network feedback study, explored two situations 1,fine tuning the whole network, 2, only adjust the full connection layer.

Test:

The first Test framework: the location network is applied only to the image's crop center and is used to compare performance under different network modifications.

1, it is found that the positioning performance of the whole network is fine-tuning, and the ratio adjustment is better than the positioning result of all connected layer weights.

2,PCR is better than SCR, and this is the opposite of Overfeat's results.

So the best way to locate is to use pcr,fine-tuning Entire network.

The second Test framework: Using Overfeat's greedy fusion process (not using offset pooling), the location network is applied intensively throughout the image, first the confidence score of bounding box is given according to the Softmax classification result, and then the space-like bounding is fused. box, and finally select the maximum confidence score of bounding box.

Under different scale, locate the result.

1, the appropriate scale has an effect on the positioning result s=384 better than s=256.

2,multi-scale is better than Single-scale.

3,multi-model Fusion will be better.

Compared with other State-of-the-art methods:

Under the use of less scale, without the use of offset pooling case, the results of this paper is much higher than overfeat; The main reasons for this improvement are the different network structure, good network, good classification, good location, and good detection.

VI, Conclusion

Depth is the key to getting good results.

Some understanding and confusion

I feel this article in the network discussion phase and blog 9 to explore the best network comparison is similar; first, a good result of the basic network A, and then on the network a some column modification, step by step to improve network performance, and then explore the good network design should be what kind of Blog 9 The Ways of balancing the network are more sophisticated than the complexity of this article. And the conclusions about depth of the two articles are the same, and the depth can increase the result, but the depth will saturate.

The biggest question here is how their network a is raised.

The A structure in post 9:

Input (224,224,3) →64f (7,7,3,s=2) →max-p (3,3,3) →128f (5,5,64) →max-p (2,2,2) →256f (3,3,128) →256f (3,3,256) →256F ( 3,3,256) →SPP (6*6,3*3,2*2,1*1) →4096fc→4096fc→softmax

I feel that the structure of blog 9 draws on the alex-net,zf-net and the vgg of this article, for example, this three-segment structure is from the thin version of the alex-net and Zf-net, the first convolutional layer has 64 filter instead of 96, it is borrowed from the first convolution layer of the structure There is also the possibility of excluding LRN based on this article or overfeat.

However, the structure of this article can not find a relationship, 5 stage, the number of filter size, network structure, such as 64-128-256-512-512; the author of the 3*3 filter has explained the reason, but the key network structure author does not give a concrete design basis, Rather, it gives the result directly; the feeling that one is by virtue of experience, and then by virtue of experimentation, combined with the point of view in post 9, the VGG network can also use its middle-level analysis and replacement ideas to improve the classification results.

One confusion is that "PCR is better than SCR, this is the opposite of Overfeat"; overfeat in the three scale SCR-VS-PCR for 31.3-vs-44.1; SCR is significantly better than the Pcr,overfeat author's explanation is that there are more regression functions at the top of the PCR, and then there are more parameters, and each class has a limited number of training samples, resulting in each category of regression function is not very well trained, but the authors of the network of PCR significantly better than SCR, Training samples did not increase, and now feel overfeat this kind of training samples of the less reasonable explanation is debatable; In addition, the different sense of network structure is the key of two result difference, except the convolution extraction stage is different The structure of the whole connection layer is also different overfeat network is 4096-1024-regression; The network of this paper is 4096-4096-regression; the cause of this disparity is estimated to be the problem of network result. But this can explain why the results of this article are better than the results of Overfeat, if the network structure to explain the PCR better than SCR, feel a bit far-fetched. So the feeling of PCR and SCR two predictive bounding box methods is still somewhat confused. "There is a different place in the above test, this article gives a real picture of the classification, and then more real analogy, predicting bounding box, rather than using the first prediction classification, in predicting the bounding box method."

Second, model analysis

It is known that the VGG is improved on the alexnet, and the comparison is as follows:

Same point

1. The last three Layer FC layer (Fully connected fully connected layer) has the same structure.
2. Are divided into five layers (groups).
3. Separate each layer and each layer with a pooling layer.

Different points

1.AlexNet Each layer contains only one convolution layer, the size of the filter 7x7 (very Large), and Vgg each layer contains multiple (four) convolution layer, the size of the filter is 3x3 (minimum). Obviously, Vgg is mimicking Alex's structure, but it does so by reducing the size of the filter and increasing the number of layers to achieve the same effect. I present one of my own understandings of this imitation. Because it is not mentioned in the paper, it is only my own understanding, for everyone's reference.

The author said a sentence in the paper
"This can is seen as imposing a regularisation on the 7x7 conv. filters, forcing them to has a decomposition through th E 3x3 Filters "
He says 7x7 filter can be decomposed into several 3x3 filter overlays.

An analogy of the orthogonal decomposition of vector x,x in n-dimensional space
x = x1 (1, 0, 0, ...) + x2 (0, 1, 0, ...) + x3 (0, 0, 1,...) + ... + xn (0, 0, 0, ..., 1)

The filter for each layer of each group is likened to the base of the n-dimensional Euclidean space.
If a group of Vgg contains a 3-layer 3x3 filter, then we assume that a 7x7 filter can be decomposed into 3 "orthogonal" 3x3 filter.

Author Original: First, we incorporate three non-linearrectification layers instead of a single one, which makes the decision function More discriminative. Second, we decrease the number of parameters:assuming that both the input and the output of the Athree-layer 3x3 Convolutio n Stack has C channels, the stack was parametrised by 3 32c^2 = 27c^2weights; At the same time, a single 7x7 conv. Layer would require 72c^2 = 49c^2

The channel of 2.AlexNet is significantly smaller than VGG. Guessing Vgg's ability to achieve higher accuracy stems from more channel numbers. As the filter size decreases, the channel can be greatly increased and more information can be extracted.

Vgg really has too little innovation, he just told us through experiments that the depth of the network has a significant impact on performance, and then is the convolution kernel size understanding.

Related references:

http://blog.csdn.net/whiteinblue/article/details/43560491

http://hacker.duanshishi.com/?p=1690

http://www.jianshu.com/p/9c6d90e4f20e

V-Bomb: Vgg