ResNet Residual Network

Source: Internet
Author: User

We have introduced the classic network in the front, we can view the previous article: Shallow into the TensorFlow 6-to achieve the classic network

With the network more and more deep, we found that only by BN, Relu, dropout and other trick can not solve the convergence problem, on the contrary, the deepening of the network to bring the increase in parameters.

Based on previous experience, we know: The network is not the deeper the better, on the one hand too many parameters easily lead to the fitting (of course, the sample enough to some extent to solve the problem); On the other hand, the training results will change around the truth, resulting in network turbulence.

Learn from previous experience, we in the training classifier, the use of GBDT and xgboost thinking, with the help of residual to solve the vibration problem .


ResNet full name is residual network, each node is no longer the parameter itself, but the residual, which determines the network may be infinitely deeper, the baseline is unchanged, the following node to learn is to the front node complement, although there are shocks, but the scope of the concussion is getting smaller, until it tends to 0.

ResNet's network structure draws on the highway, adds a path from input to output, that is shortcut connection, which is the key point.


The module is described by a formula:


For input x, the parameter is learned by the intermediate function f (x,w), and F is the residual function we say.

The middle of the module corresponds to two Layer, each Layer processing a multiplication and a relu,f (x) +x is based on X to achieve a fast connection, after the connection to implement a relu.

The first resnet was a 152-story network presented by the MSRA team, which won the championship in the 2015 ILSVRC2015, to see the comparison:


than the 14 Vgg and googlenet a lot better, error rate of 3.57 quite eye-catching, its network structure is what kind of it. Do not think, must be a big string (all greedy bar.) )


A large number of block units, the current several commonly used ResNet networks include: resnet-50/101/152, of course, the more the number of layers of computation, based on resnet improvements we also know ahead of time, including Google's Inception-resnet-v2, kaiming, Resnext and so on.

OK, we'll introduce resnext in the next article.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.