Densenet----Narrowing the network

Densenet's idea largely stems from the work that we published last year on ECCV called a random depth network (deep networks with stochastic depth). At that time we proposed a method similar to dropout to improve the resnet. We find that each step of the training process randomly "throws" some layers, which can significantly improve the generalization performance of ResNet. The success of this approach brings us at least two points of inspiration:

First, it shows that neural networks do not necessarily have a progressive hierarchy, which means that a layer in a network can rely on the characteristics of a more front-level learning, not just the next-level feature. Imagine that in a random depth network, when the L layer is thrown away, the l+1 layer is directly connected to the L-1 layer, and when the 2nd to the L layer is thrown away, the l+1 layer directly uses the characteristics of the 1th layer. Therefore, the random depth network can actually be regarded as a densenet with random dense connection.

Secondly, we randomly throw away many layers in the course of training without destroying the convergence of the algorithm, which shows that the ResNet has obvious redundancy, and each layer in the network only extracts a few features (the so-called residuals). In fact, we have trained ResNet randomly removed a few layers, the prediction of the network will not have a great impact on the results. Since the characteristics of each layer of learning so little, can you reduce its computational capacity to reduce redundancy?

Densenet's design is based on the above two observations. We make each layer of the network directly connected to its front layer, to achieve the reuse of features, and at the same time, each layer of the network is designed to be particularly "narrow", that is, only to learn very few feature maps (the most extreme case is that each layer only learns one feature map) to reduce redundancy. These two points are also the main difference between densenet and other networks. It should be stressed that 1th is the 2nd premise, there is no dense connection, we are not able to design the network too narrow, otherwise training will appear under the under-fitting phenomenon, even if ResNet.

Densenet----Narrowing the network