Applied Deep Learning Resources

Applied Deep Learning Resources

A Collection of articles, blog posts, slides and code snippets about deep learning in applied settings. Including trained models and simple methods The can is used out of the box. Mainly focusing on convolutional neural Networks (CNN) But recurrent neural Networks (RNN), Deep Q-networks (DQN) and othe R interesting architectures'll also be listed.

Cnn

Latest Overview of the CNNs can is found from the paper ' Deep Learning for Visual understanding:a review ' [link, PDF]

Another decent overview in Nature by LeCun, Bengio and Hinton: "deep learning" [link, PDF]

ImageNet

ImageNet is the most important image classification and localization competition. Other data sets with results can is found from here: "Discover the current state of the art in Objects classification. " [link].

Prediction error of the ImageNet competition have been decreasing rapidly over the last 5 years:

Main Network architectures on Imagenetalexnet

Original paper: "ImageNet classification with deep convolutional neural Networks" [PDF]

Properties:8 Weight layers (5 convolutional and 2 fully connected), million parameters, rectified Linear Units (relus) , Local Response Normalization, dropout

Vgg

Original paper: "Very deep convolutional Networks for large-scale Image recognition" [arxiv]

Properties:19 weight layers, 144m parameters, 3x3 convolution filters, L2 regularised, dropout, No Local Response Normali Zation

Googlenet

Original paper: "going deeper with convolutions" [arxiv]

Lates upgrade to the model achieves even better scores with models and import to Torch: "Rethinking the Inception Arch Itecture for computer Vision " [arxiv], " Torch Port of Inception V3 " [GitHub]

properties:22 layers, 7m parameters, Inception modules, 1x1 conv layers, relus, dropout, mid-level outputs

Inception Modules:

ResNet

Original paper: "deep residual learning for Image recognition" [arxiv]

Very Nice slides: "deep residual learning" [PDF]

GitHub: [GitHub]

properties:152 layers, Relus, batch normalization (see) batch normalization:accelerating deep Network Training by Re ducing Internal covariate Shift " [arxiv]), less hacks (no dropout), + stable (different number of layers work as W ELL) and lower complexity than VGG.

Main building block of the network:

Features is also very good and transferable with (faster) R-cnns (see below):

Other architectures

• Deep Learning for 3D shapes: "3D shapenets:a deep representation for volumetric shapes" [PDF]

• Code and a model for faces: "free and open source face recognition with deep neural networks." [GitHub]

• Fast Neural networks which can perform arbitrary filters for images: "deep edge-aware Filters" [PDF]

• Lot's of different models in Caffe's "Model Zoo" [GitHub]

Feature Learning and Object detection

• "CNN Features off-the-shelf:an astounding Baseline for recognition" [arxiv]

• First paper about R-CNN: "Rich feature hierarchies for accurate object detection and semantic segmentation" [PDF, Slides

• "Faster r-cnn:towards real-time Object Detection with region proposal Networks" [ArXiv, GitHub, Slides]

• "An empirical Evaluation of deep learning on Highway Driving" [arxiv]

• "Object detectors Emerge in deep Scene CNNs" [arxiv]

• Faster and better features: "Efficient deep Feature learning and Extraction via Stochasticnets" [arxiv]

Other

• Code and models for automatic captions of images: "deep visual-semantic alignments for generating Image descriptions" /c1>[web poster, PDF, GitHub]

• Google deep Dream or neural networks in LSD: "inceptionism:going deeper into neural networks" [Link,deepdreamer. io/]

Deep Dreaming from Noise:

• Code and models for mixing arbitrary content and art style: "A neural algorithm of artistic style" [arxiv, Deepar T.io, a blog post, Github.com/jcjohnson/neural-style]

• "Automatic colorization" and it includes a pre-trained model [Link]

• "Learning visual similarity for product design with convolutional neural Networks" [PDF]

• Using images and image descriptions to improve search results: "images Don t lie:transferring deep Visual Semantic Fe Atures to large-scale multimodal learning to Rank " [arxiv]

• "How Google Translate squeezes deep learning onto a phone" [post]

• "What a deep neural Network thinks about your #selfie" [Blog]

Top selfies According to the convnet:

• "recommending music on Spotify and deep learning" [GitHub]

• "deepstereo:learning to Predict New views from the world ' s Imagery" [arxiv]

• Classifying street signs: "The power of spatial Transformer Networks" [blog] with "spatial Transformer netwo Rks " [arxiv]

• "Pedestrian Detection with RCNN" [PDF]

Dqn

• Original paper: "Playing Atari with deep reinforcement learning" [arxiv]

• My Popular Science article on DQN: "Artificial General Intelligence the plays Atari video Games:how did DeepMind Do it? " [link]

• DQN for RoboCup: "deep reinforcement learning in parameterized Action Space" [arxiv]

RNN

• Original paper of the best RNN Architecture: "Long Short-term Memory" [PDF]

• Very Good tutorial-like Introduction to Rnns by Andrej Karpathy: "The unreasonable effectiveness of recurrent neural N Etworks " [link]

• "Visualizing and understanding recurrent Networks" [arxiv]

• "Composing Music with recurrent neural Networks" [Blog]

Other promising or useful architectures

• HTMS by Jeff Hawkins: "continuous online sequence learning with an unsupervised neural network model"? [arxiv]

• Word2vec: "Efficient estimation of Word representations in Vector Space" [arxiv, Google code]

• "Feedforward sequential Memory networks:a New Structure to learn long-term Dependency" [arxiv]

Framework Benchmarks

• "Comparative Study of Caffe, Neon, Theano and Torch for deep learning" [arxiv]

Their summary:from our experiments, we observe that Theano and Torch is the most easily extensible frameworks. We observe that Torch are best suited for any deep architecture on CPUs, followed by Theano. It also achieves the best performance on the GPU for large convolutional and fully connected networks, followed closely by Neon. Theano achieves the best performance on GPUs for training and deployment of LSTM networks. Finally Caffe is the easiest for evaluating, the performance of the standard deep architectures.

• Very good qualitative analysis:zer0n/deepframeworks: [GitHub]

• Just performance Comparison:soumith/convnet-benchmarks: [GitHub]

• "Deep learning Libraries by Language" [link]

Other resources

• "An awesome list of (large-scale) public datasets on the Internet. (on-going collection) " [GitHub]

• "Model Zoo" [GitHub]

• Videos from "deep learning Summer School, Montreal": http://videolectures.net/deeplearning2015_montreal/

• http://deeplearning.net/

Credits

Most of the snippets has come to my attention via internal mailing lists of computational Neuroscience labs at University of Tartu and london-based visual search company Dream it Get it. I am also reading a weekly newsletter by Data Elixir and checking I papers of the The both main deep learning conference S:ICML and NIPS.

• Contact GitHub
• Api
• Training
• Shop
• Blog
• About

• ©2016 GitHub, Inc.
• Terms
• Privacy
• Security
• Status
• Help

Applied Deep Learning Resources