Deep Learning Series (4): Sparse Coding and ICA)

I 've been hesitant about How to Write sparse encoding, and I 've looked back and forth several times at ufldl. This is not only an important concept of DL deep learning, but also the pillar of stacked ISA deep feature learning that I have been studying for a long time.

This chapter mainly introduces the main concepts of sparse encoding and the principal component analysis method.

1. Sparse Coding ):

The Sparse Coding Algorithm is an unsupervised learning method. It is used to find a group of "super-complete" base vectors to more efficiently represent sample data. (If the dimension of X is N, then K> N)

The ultra-complete foundation can more effectively find the structures and patterns implied in the input data. However, if coefficient A is no longer uniquely determined by X, then "sparsity" is added to solve the degradation problem caused by over-completeness.

For example:

 

 

"Sparsity" is only a few non-zero elements or only a few elements far greater than 0.

The sparse encoding cost function of M input vectors is defined:

Among them, S (.) is a sparse cost function that "punishes" a greater than 0 ". Generally, select the paradigm cost function S (A) = | A | and the logarithm cost function S (A) = Log (1 + a ^ 2 ). generally, in order to reduce the number of A or increase the number of PHI and reduce the sparse penalty, the limit is | Phi | ^ 2 less than the constant C.

 

The method for learning the base vector set is 1. Use training sample x one by one to optimize the coefficient A, 2. process multiple samples at a time to optimize the base vector Phi.

Limitations: Even if you have learned to get a group of base vectors, if you want to "encode" the new data samples, you must re-execute the optimization process to obtain the required coefficients, increasing the computing cost.

Ii. Principal Component Analysis (ICA)

 

The base vectors obtained by sparse encoding are not necessarily linear independent. The basis of independent principal component analysis (ICA) learning is not only linear but also standard orthogonal (Phi T = I)

Here, ICA mainly learns a group of base vectors W, and maps X to features through W. features are sparse. (Opposite to Sparse Coding ).

The objective functions of standard orthogonal ICA are: (two expressions)

Minimize J (w) = | (| wx |) |

S. T. ww '= I

The constraint WW '= I implies two other constraints:

1. The number of base vectors must be smaller than the input data dimension (incomplete), because we need to learn a set of standard orthogonal basis.

2. The data must have no regular zca whitening. (Normalization is worth noting !)

To optimize the target function, you can use the gradient descent method and add the projection steps in each step:

 

However, In the ufldl tutorial deriving gradientsusing the Backpropagation idea, the re-built cost function is:

Apply back propagation (back to reverse transmission of ufldl)

Partial derivation formula:

The orthogonal formula of W is:

For the implementation code, see the ICA model exercise in tornadomeet's blog. (Changed)

And then upload the relevant code.

 

Refer:

1. tornadomeet blog

2. Dark's blog

3. Search for foxes (normalization and normalization)

4. ufldl tutorial

5. The document natural image statistics (aapo hyv ○ arinen) about ICA is very long and classic. It is worth reading!

Updating .... (PS: should the Edit function of csdn be upgraded! You cannot edit or copy formulas, which is quite troublesome !)