Deep Learning (depth learning) Learning Notes finishing Series (v)

Deep Learning (depth learning) Learning notes finishing Series

[Email protected]

Http://blog.csdn.net/zouxy09

Zouxy

Version 1.0 2013-04-08

Statement:

1) The Deep Learning Learning Series is a collection of information from the online very big Daniel and the machine learning experts selfless dedication. Please refer to the references for specific information. Specific version statements are also referenced in the original literature.

2) This article is for academic exchange only, non-commercial. So each part of the specific reference does not correspond in detail. If a division accidentally violated the interests of everyone, but also look haihan, and contact bloggers deleted.

3) I Caishuxueqian, finishing summary of the time is inevitable error, but also hope that the predecessors, thank you.

4) Reading this article requires machine learning, computer vision, neural network and so on (if not, it doesn't matter, no see, can read, hehe).

5) This is the first version, if there are errors, you need to continue to amend and delete. Also hope that we have a lot of advice. We all share a little, together for the promotion of the Motherland Scientific research (hehe, good noble goal ah). Please contact: [Email protected]

Directory:

I. Overview

Second, the background

III. visual mechanism of human brain

Iv. about Features

4.1, the granularity of the characteristic representation

4.2. Primary (shallow) feature representation

4.3, structural characteristics of the expression

4.4. How many features are needed?

The basic thought of deep learning

Vi. Shallow learning (shallow learning) and deep learning (Deepin learning)

Seven, deep learning and neural Network

Eight, deep learning training process

8.1. Training methods of traditional neural networks

8.2. Deep Learning Training Process

Common models or methods of deep learning

9.1, Autoencoder Automatic encoder

9.2, Sparse coding sparse coding

9.3. Restricted Boltzmann Machine (RBM) restricts the Boltzmann machines

9.4, deep Beliefnetworks convinced that the degree of network

9.5. Convolutional Neural Networks convolutional neural network

Ten, summary and Prospect

Xi. bibliography and deep Learning learning resources

Pick up

9.2, Sparse coding sparse coding

If we loosen the output to be equal to the input and take advantage of the concept of the base in linear algebra, i.e. o = a1*φ1 + a2*φ2+....+ an*φn,φi is the base, and AI is the coefficient, we can get an optimization problem:

Min | i–o|, where I is the input and O represents the output.

By solving this optimization equation, we can obtain the coefficients AI and the base φi, which are the other approximate expressions of the input.

Therefore, they can be used to express input I, and this process is also learned automatically. If we add the L1 regularity limit to the above equation, we get:

Min | i–o| + u* (|a1| + |a2| + ... + |an |)

This method is called sparse Coding. In layman's words, a signal is expressed as a set of linear combinations of bases, and the signal can be expressed by requiring fewer bases. "Sparsity" is defined as: only a few non-0 elements or only a few elements far greater than 0. The requirement factor AI is sparse meaning that: for a set of input vectors, we only want to have as few coefficients as possible far greater than 0. There is a reason to choose to use the sparse component to represent our input data, because the vast majority of sensory data, such as natural images, can be represented as a superposition of a small number of basic elements that can be polygons or lines in an image. At the same time, the analogy with the primary visual cortex has thus been improved (the human brain has a large number of neurons, but for some images or edges there are only a few neuron excited, others are suppressed).

Sparse coding algorithm is a unsupervised learning method, which is used to find a set of "super-complete" base vectors to represent the sample data more efficiently. Although the form of principal component Analysis (PCA) allows us to conveniently find a set of "complete" base vectors, what we want to do here is to find a set of "super-complete" base vectors to represent the input vectors (that is, the number of base vectors is greater than the number of dimensions of the input vectors). The advantage of super-complete base is that they can more effectively find out the structure and pattern implicit in the input data. However, for ultra-complete bases, the coefficient AI is no longer determined by the input vectors alone. Therefore, in the sparse coding algorithm, we add a criterion of "sparsity" to solve the degradation (degeneracy) problem caused by super-completeness. (Please refer to the detailed procedure: UFLDL Tutorial Sparse coding)

For example, in the image of the feature extraction to do the edge detector generation, then the job here is images from the natural randomly pick some small patches, through these patches generated to describe their "base", That is, the right side of the 8*8=64 a basis composed of basis, and then given a test patch, we can follow the above formula through the linear combination of basis, and sparse matrix is a, in a, there are 64 dimensions, of which 0 is only 3, it is called " Sparse ".

There may be questions here, why the bottom as edge detector? What about the upper deck? Here is a simple explanation we will understand that the reason is edge detector because the edge in different directions can describe the entire image, so the edge of different directions is naturally the image of the basis ... And the result of the basis combination on the upper layer is the upper layer of the combination basis ... (That's what we said in the fourth part above)

The Sparse coding is divided into two parts:

1) Training stage: given a series of sample images [X1, x 2, ...], we need to learn to get a set of bases [Φ1,φ2, ...], which is the dictionary.

Sparse coding is a variant of the K-means algorithm, and its training process is similar (the idea of the EM algorithm: If the objective function to be optimized consists of two variables, such as L (W, B), then we can first fix W, adjust B to make L minimum, then fix B, adjust W to make l minimum, such iterative alternating, Continuously pushes l to the minimum value. The EM algorithm can be seen in my blog: "From the maximum likelihood to the EM algorithm Shallow solution").

The training process is a repetitive iterative process, as described above, our alternating changes of a and φ make the following objective function minimal.

Each iteration is divided into two steps:

A) fixed dictionary φ[k], then adjust a[k], so that the above, that is, the objective function is the smallest (that is, the solution lasso problem).

b) then fix a [K], adjust φ[k], so that the upper, that is, the objective function is the smallest (that is, the problem of convex qp).

Continue to iterate until it converges. This allows you to get a set of bases that are good at representing this series of x, which is the dictionary.

2) Coding stage: given a new picture x, the dictionary obtained by the above gets the sparse vector aby solving a lasso problem. This sparse vector is a sparse representation of this input vector x.

For example:

Continuation of

Deep Learning (depth learning) Learning Notes finishing Series (v)