The basis of theoretical interpretation of deep learning

Reference documents:

Feature Extraction:

In deep learning, the amount of information that the lower layer carries is greater than the amount of information on top . The lowest layer is considered a base. For example, in high-dimensional space, there is always a set of complete bases. Any vector can be represented by a complete base line. This is, after a multilayer representation, the rank of the matrix of the subsequent vectors is less than or equal to the rank of its next layer of vector composition matrix of course, when we first introduced here, we also thought that any graph could be represented as a linear combination of 400 graphs.

However, the layers in the actual deep learning have some activation functions, which are non-linear. If so, the purple part should be changed to " any layer s and the first layer of mutual information less than equals s next layer with the first layer of information ." Note that this must be mutual information with the first layer. This additional information may be passed on to the next layer because there may be additional information added to each layer, in addition to some of the information being cut. This leads to a big problem if you compare only with any neighboring mutual information.

Shallow-level learning:

In the the late 1980s, neural networks began to appear. However, the neural network at this time is just a layer of hidden layers.

The 1990s, various can be regarded as shallow network successive appearance: Svm,boosting, Maximum entropy method. At this point, the shallow neural network is caught in silence.

2006,geoffrey Hinton and his student Ruslansalakhutdinov published an article in science that opened the deep learning craze. Its main ideas:

1) The artificial neural network of the multiple hidden layer has excellent characteristic learning ability, and the learning features have a more essential characterization of the data, which is beneficial to visualization or classification.

2) The difficulty of the deep neural network in training can be effectively overcome by the "layer initialization" (Layer-wise pre-training), in this article, the level-by-layer initialization is realized by unsupervised learning.

Disadvantages:

1) Difficult to express complex classification mapping, feeling under-fitting, low generalization ability. (Of course, the depth of the study, the generalization ability will be low, then controllable)

Deep Learning:

In order to solve the shallow learning due to the lack of generalization ability caused by low, naturally think of complex mapping. What kind of complexity? Just choosing an elementary function is far from enough. How about choosing a combination of elementary functions? How is the group legal? Linear addition? One as the other index? There are such considerations, such as:

In fact, we have a classic mapping that has been tested for good results.

Advantages:

Different from the traditional shallow learning, the difference of deep learning is that: 1) emphasizes the depth of the model structure, usually has 5 layers, 6 layers, or even 10 layers of hidden layer nodes; 2) clearly highlights the importance of feature learning, that is to say, by changing the characteristics of the original space to a new feature space, This makes it easier to classify or predict. Compared with the method of constructing characteristics of artificial rules, the use of big data to learn the characteristics, more able to depict the rich intrinsic information of the data.

The basis of theoretical interpretation of deep learning