Mxnet, Caffe, TensorFlow, Torch, Theano

http://blog.csdn.net/myarrow/article/details/52064608
1. Basic Concepts 1.1 Mxnet Related Concepts

Deep learning goals: How to express neural networks in a convenient way, and how to train quickly to get models

CNN (convolution layer): Expressing spatial relevance (learning representation)

Rnn/lstm: Expression time continuity (modeling timing signal)
Imperative programming (imperative programming): Shallow embedding, where each statement is executed in the original sense, such as numpy and torch

Declarative language (declarative programing): Embedded deep, providing a complete set of mini languages for specific applications. That is, the user only needs to declare what to do, and the concrete execution is done by the system. Such systems include Caffe,theano and TensorFlow. Imperative programming is obviously easier to understand some, more intuitive some, but declarative more conducive to doing optimization, and more conducive to do automatic derivation, so all keep.

	Shallow embedding, command-type programming	Deep embedding, declarative programming
How to perform a=b+1	Need B has been assigned value. Performs the addition immediately, saving the results in a.	Returns the corresponding calculation graph (computation graph), which we can then assign to B and then perform the addition operation
Advantages	Semantically easy to understand, flexible, and can precisely control behavior. It is often possible to seamlessly interact with the main language, using the main language's various algorithms, toolkits, bugs, and performance debuggers.	We've got the whole diagram when we really started the calculation, so we can do a series of optimizations to improve performance. It is also easy to implement auxiliary functions, such as providing forward and backward functions for any calculation diagram, visualizing the diagram, saving the graph to the hard disk and reading from the hard disk.
Disadvantages	It is difficult to implement unified auxiliary functions and provide overall optimization.	Many of the main language features are not used. Some are simple to implement in the main language, but are often troublesome here, such as if-else statements. Debug is not easy, for example, monitoring the intermediate results of a node in a complex calculation diagram is not easy.

At present, most of the existing systems adopt one of the two programming modes. Unlike them, Mxnet tries to combine the two patterns seamlessly. Mxnet provides tensor operations on imperative programming, while mxnet supports symbolic expressions in declarative programming. Users can freely mix them to quickly realize their ideas. For example, we can use declarative programming to describe neural networks and train models using automated derivation provided by the system. Another convenience, the model of iterative training and updating Model Law may involve a lot of control logic, so we can use imperative programming to achieve. At the same time we use it for convenient mode and interactive data with the main language.

1.2 Key features of depth learning

(1) Hierarchy abstraction

(2) End to end learning 2. Comparison Table

Compare items	Caffe	Torch	Theano	TensorFlow	Mxnet
Main language	C++/cuda	C++/lua/cuda	Python/c++/cuda	C++/cuda	C++/cuda
From the language	Python/matlab	-	-	Python	Python/r/julia/go
Hardware	Cpu/gpu	Cpu/gpu/fpga	Cpu/gpu	Cpu/gpu/mobile	Cpu/gpu/mobile
Distributed	N	N	N	Y (not open source)	Y
Speed	Fast	Fast	Medium	Medium	Fast
Flexibility	So so	Good	Good	Good	Good
Document	Comprehensive	Comprehensive	Medium	Medium	Comprehensive
Fit model	Cnn	Cnn/rnn	Cnn/rnn	Cnn/rnn	Cnn/rnn?
Operating system	All Systems	Linux, OS X	All Systems	Linux, OS X	All Systems
Command-	N	Y	N	N	Y
Declaration type	Y	N	Y	Y	Y
Interface	Protobuf	Lua	Python	C++/python	Python/r/julia/go
Network structure	Layered approach	Layered approach	Symbol tensor chart	Symbol tensor chart	?

Note: 1) using symbolic Zhang Shitu to describe the model, adding new layers is more convenient, while layered methods add new layers that need to be implemented themselves (Forward,backward and gradient update functions). 3. Detailed Description 3.1 mxnet

The Mxnet system architecture is shown in the following illustration:

From top to bottom for a variety of main language embedding, programming interface (matrix operation, symbolic expression, distributed communication), two programming modes of unified system implementation, as well as the hardware support.

The design details of mxnet include: symbol execution and automatic derivation; running dependent engine; Memory savings. 3.2 Caffe

Advantages:

1 The first mainstream industry-level depth learning tool.
2 It began at the end of 2013, by the UC Berkely yangqing Jia teachers to write and maintain an excellent convolution neural network implementation. Caffe is still the most popular tool pack in the field of computer vision.

3) specializes in image processing

Disadvantages:

1 It has many extensions, but because of some legacy architectural problems, it is inflexible and has poor support for recursive networks and language modeling.

2 based on the network structure of the layer, its expansibility is not good, for the newly added layer, it needs to implement itself (forward, backward and gradient upda