ResNet Residual Network

We have introduced the classic network in the front, we can view the previous article: Shallow into the TensorFlow 6-to achieve the classic network

With the network more and more deep, we found that only by BN, Relu, dropout and other trick can not solve the convergence problem, on the contrary, the deepening of the network to bring the increase in parameters.

Based on previous experience, we know: The network is not the deeper the better, on the one hand too many parameters easily lead to the fitting (of course, the sample enough to some extent to solve the problem); On the other hand, the training results will change around the truth, resulting in network turbulence.

Learn from previous experience, we in the training classifier, the use of GBDT and xgboost thinking, with the help of residual to solve the vibration problem .

ResNet full name is residual network, each node is no longer the parameter itself, but the residual, which determines the network may be infinitely deeper, the baseline is unchanged, the following node to learn is to the front node complement, although there are shocks, but the scope of the concussion is getting smaller, until it tends to 0.

ResNet's network structure draws on the highway, adds a path from input to output, that is shortcut connection, which is the key point.

The module is described by a formula:

For input x, the parameter is learned by the intermediate function f (x,w), and F is the residual function we say.

The middle of the module corresponds to two Layer, each Layer processing a multiplication and a relu,f (x) +x is based on X to achieve a fast connection, after the connection to implement a relu.

The first resnet was a 152-story network presented by the MSRA team, which won the championship in the 2015 ILSVRC2015, to see the comparison:

than the 14 Vgg and googlenet a lot better, error rate of 3.57 quite eye-catching, its network structure is what kind of it. Do not think, must be a big string (all greedy bar.) ）

A large number of block units, the current several commonly used ResNet networks include: resnet-50/101/152, of course, the more the number of layers of computation, based on resnet improvements we also know ahead of time, including Google's Inception-resnet-v2, kaiming, Resnext and so on.

OK, we'll introduce resnext in the next article.