"Cs229-lecture14" Singular value decomposition

There are two general implementations of PCA, one is realized by eigenvalue decomposition, and one is realized by singular value decomposition.

Content:

• PCA (principal component analysis) is a direct dimensionality reduction method, by solving eigenvalues and eigenvectors, and selecting some characteristic vectors with large eigenvalues to achieve the effect of dimensionality reduction.
• One application of PCA--lsi (latent Semantic indexing, implied semantic index);
• An implementation of PCA--SVD (Singular value decomposition, singular value decomposition);
• ICA (independent component analysis)

Implicit semantic index (LSI)

What does LSI mean?

The so-called recessive semantic index refers to how to find out the relationship between words through massive literature. When two words or a group of words appear in large numbers in the same document, the words can be considered semantically related. The machine does not know what a word actually stands for, and does not know what a word means.
Like what:
(1) Computer and computer These two words when people write articles often mixed, these two words in a large number of pages in the same time, the search engine will think that these two words are extremely semantic related. (2) SEO and search engine optimization (although one is English, one is Chinese) These two words in the same page, although the search engine can not know what SEO or seo refers to what, but can be semantically "seo", "Search Engine optimization", "search engine Optimization "," SEM "and other words tightly linked together. Visual potential semantic indexing is not dependent on language.
(3) Two words, such as apples and oranges, also appear in large numbers in the same document, but are less dense than synonyms. So search engines don't think of them as semantically relevant.

Vectors represent text, and dimensions are high. So what is implicit semantic indexing? Implicit semantic indexing in principal component analysis means that words with the same meaning are mapped to the same dimension in the low-dimensional space through dimensionality reduction. This can reduce computational complexity and reduce noise. Among them, the reduction of noise refers to two of completely non-similar texts on high dimensions that may become similar after dimensionality reduction. For example, if there is an article that contains only one word learn, and the other article contains only study, the computation of similarity on the high dimension is 0, and the Learn and study are mapped to the same dimension by means of implicit semantic index, then the similarity is more accurate.

SVD (Singular value decomposition, singular value decomposition)

Transferred from: http://blog.csdn.net/ningyaliuhebei/article/details/7104951

Singular value decomposition is a very obvious physical meaning of a method, it can be a more complex matrix with smaller and simpler several sub-matrices of the multiplication to represent, these small matrices describe the matrix of important characteristics. It is like describing a person, to describe to others that this person is bushy, square face, beard, and with a black frame of the glasses, such a few characteristics, let others mind inside there is a more clear understanding, in fact, the characteristics of human face is an infinite variety of, the reason can be described, Because people are born with a very good ability to extract important features, so that the machine learns to extract important features, SVD is an important method.

In the field of machine learning, there are quite a number of applications and singular values can be related, such as feature reduction PCA, do data compression (image compression as the representative) of the algorithm, as well as search engine semantic level search LSI (latent Semantic Indexing)

The following transfers are from: http://blog.csdn.net/wangzhiqing3/article/details/7446444

Basic knowledge

1. Rank of matrix: the rank of the matrix is the number of linearly unrelated rows or columns in the matrix

2. Diagonal matrix: The diagonal matrix is a square that all elements are zero except the diagonal

3. Unit matrix: If all diagonal elements on the diagonal matrix are 1, the matrix is called the unit matrix

4. Eigenvalues: To a m x m matrix C and vector x, if present λ makes the following form

is called λ is the eigenvalues of matrix C, X is called the Matrix eigenvector. The number of non-0 eigenvalues is less than or equal to the rank of the matrix.

5. Relationship between eigenvalues and matrices: consider the following matrices

The matrix eigenvalue λ1 = 30,λ2 = 20,λ3 = 1. corresponding feature vectors

Suppose vt= (2,4,6) calculates S x VT

It can be seen from the above results that the result of multiplying the matrix with the vector is related to the eigenvalues and eigenvectors. Observe the three eigenvalues λ1 = 30,λ2 = 20,λ3 = 1,λ3 value is the least, the effect on the calculation results is also minimal, if the λ3 is omitted, then the result of the operation is equivalent to (60,80,6) to (60,80,0), the two vectors are very similar. This also indicates that the small value of the eigenvalues of the matrix-vector multiplication results in a small contribution, small impact. This is also the mathematical basis for the low-order approximation that is discussed later.

Matrix decomposition

1. Decomposition of square matrices

1) Set S is m x m phalanx, there is the following matrix decomposition

where u is listed as S eigenvector, which is the diagonal matrix, where the value on the diagonal is the eigenvalues of S, which are arranged from large to small:

2) set S is the M x m phalanx and is a symmetric matrix with M eigenvectors. The following decomposition is present

where Q is listed as the matrix S of the unit orthogonal eigenvector, still represents the diagonal matrix, where the value on the diagonal is the characteristic value of S, in order from large to small. Finally, qt=q-1, because the inverse of the orthogonal matrix equals its transpose.

2. Singular value decomposition

The decomposition of the square is discussed above, but in the LSA we are going to decompose the term-document matrix, it is obvious that this matrix is not a square. Singular value decomposition is required to decompose the term-document. The inference of singular value decomposition uses the decomposition of the square matrix described above.

Suppose C is a m x n matrix, U is a m x m matrix, where U is listed as an orthogonal eigenvector of CCT, V is an n x n matrix, where V is listed as the orthogonal eigenvector of CTC, and if R is the rank of C matrix, there is singular value decomposition:

Where CCT and CTC have the same eigenvalues, for

Σ is M X N, where the remaining position value is 0, and the values are sorted in descending order of size. The following is a complete mathematical definition of σ:

Ōi is called the singular value of matrix C.

Multiply the C by its transpose matrix CT to be:

The above formula is the decomposition of the symmetric matrix discussed in the previous section.

A graphical representation of singular value decomposition:

You can see that σ is a m x n matrix, but from line n+1 to M line is all zero, so it can be represented as n x n matrices, and because the right-hand matrix is multiplied so that u can be represented as M x n matrix, VT can be represented as N x n matrix

3. Low-order approximation

In LSA latent semantic analysis, low order approximation is used to represent a high-dimensional matrix using a low-dimensional matrix, and to make the difference between them as small as possible. This section focuses on low-order approximations and F-norm.

Given a m x n matrix C (whose rank is R) and a positive integer k, we would like to find a m x n matrix ck whose rank is not greater than K. Set X to the difference between C and Ck, and the F-norm of the X=c–ck,x

When K is much smaller than R, the CK is called the low-order approximation of C, where x is the difference between the two matrices as small as possible f-norm.

SVD can be used with the lower order approximation problem, the steps are as follows:

1. Given a matrix C, decomposition of its singular value:

2. Construction, it is to set the k+1 line to M line to zero, that is, the smallest r-k (the r-k smallest) singular value is set to zero.

3. Calculate CK:

Recall in the basic knowledge section once said that the value of the eigenvalues of the size of the matrix-vector multiplication effect is proportional to the size, and the singular value and eigenvalue is also proportional to the relationship, so here to choose the smallest r-k eigenvalues set to zero is reasonable, that is, we want to c-ck as small as possible. The complete proof can be found in introduction to information retrieval[2].

We now also know the basic idea of LSA: LSA wants to remove the "noise" in space by reducing the dimensions of the traditional vector space, and the dimensionality can be realized by SVD, so the term-document matrix is first SVD decomposed, then the dimensionality is reduced and the semantic space is constructed.

At the end of the SVD, Ng sums up a table.

The contents of the table are well understood, and Ng specifically emphasizes this way of thinking, looking for the same points in the algorithm and not
The same point facilitates better understanding of the algorithm.

ICA (independent component analysis)

2. ICA uncertainty (ICA ambiguities)

Since both W and s are uncertain, these two related parameters cannot be determined at the same time without a priori knowledge. For example, the above formula S=WX. When W expands twice times, s only needs to expand twice times at a time, the equation still satisfies, and therefore cannot get the unique S. At the same time, if the number of people is scrambled into another order, such as the number of the blue node becomes 3,2,1, then only a column vector order can be swapped, so it is not possible to determine the S. These two conditions are known as the original signal is indeterminate.

There is also the case that ICA is not applicable, that is, the signal can not be Gaussian distribution . Assuming that only two people emit sound signals that conform to a multivalued normal distribution, I is the unit matrix of 2*2, and the probability density function of s is needless to say, with the mean value 0 as the center, the projection surface is the peak shape of the ellipse (see multi-valued Gaussian distribution). Because, therefore, X is also Gaussian distributed, the mean value is 0, and the covariance is.

Make r an orthogonal array. If a is replaced by a '. So. The S distribution does not change, so X ' is still the mean value of 0, covariance.

Therefore, regardless of whether the mixed matrix is a or a ', the distribution of x is the same, then the mixed matrix cannot be determined and the original signal cannot be determined. 3. Density functions and linear transformations

Before discussing ICA-specific algorithms, let's review the knowledge in probability and linear algebra.

Suppose our random variable s has the probability density function (the continuous value is the probability density function, the discrete value is the probability). For simplicity, let's assume that s is a real number, and that there is a random variable x=as,a and x are real numbers. The order is the probability density of x, so how to ask?

First, the equation is transformed into, and then obtained, the solution is complete. Unfortunately, this method is wrong. For example, if s is consistent with uniform distribution (), then the probability density of s is, now make a=2, that is, x=2s, that is, X is evenly distributed on [0,2]. However, the preceding deduction will be obtained. The correct formula should be

Derivation method

More generally, if s is a vector, a reversible phalanx, then the upper formula is still set. 4. ICA algorithm

The ICA algorithm is attributed to Bell and Sejnowski, where maximum likelihood estimation is used to interpret the algorithm, and the original paper uses a complex method Infomax principal.

We assume that each has a probability density, so the joint distribution of the original signal at a given moment is

This formula represents a hypothetical premise: each person emits a sound signal independently. With P (s), we can obtain P (x)

The left side is the probability of each sampled signal x (n-dimensional vector), and the right is the product of each original signal probability | w| times.

As mentioned earlier, if there is no prior knowledge, we cannot obtain W and S. So we need to know that we're going to pick a probability density function to assign to s, but we can't pick the Gaussian distribution density function. In probability theory, we know that the density function p (x) is obtained by derivation of the cumulative distribution function (CDF) F (x). F (x) to meet two properties is: monotonically incrementing and in [0,1]. We find that the sigmoid function is well suited for defining a domain negative infinity to positive infinity, a range of 0 to 1, and a slow increment. We assume that the cumulative distribution function of S is in accordance with the sigmoid function

After derivation

This is the density function of S. Here S is the real number.

If we know the distribution function of S in advance, then we don't have to assume it, but in the case of missing, the sigmoid function can achieve good results on most problems. Since the e[s]=0 is a symmetric function, so the mean value of the e[x]=e[as]=0,x is 0, then the average of this is 0.

Yes, there's only W left. Given the sample training sample, the logarithm likelihood estimate of the sample is as follows:

Using the probability density function of the x obtained earlier, the

Curly braces are inside.

The next is the derivation of W, and here's a question of determinant | W| is the method of derivation, which belongs to matrix calculus. First, the results are given, and the derivation formulas are given at the end of the article.

The resulting derivation formula is as follows, and the derivative is (can be verified by itself):

Among them is the gradient rise rate, which is artificially specified.

Once the iteration has been found, it can be obtained to restore the original signal.

Note: when we calculate the maximum likelihood estimate, it is assumed that it is independent from the other, but this hypothesis cannot be established for speech signals or for other time-dependent features such as temperature. However, when the data is long enough, it is assumed that independence has little effect on the effect, and the convergence speed can be accelerated if the sample is scrambled beforehand and the random gradient rise algorithm is run.

Recalling the cocktail party issue, S is the signal that the person sends, is the continuous value, different time points of the s different, each person sends the signal between independent (and independent). The cumulative probability distribution function of S is the sigmoid function, but everyone emits a sound signal that matches the distribution. A (inverse of W) represents the position change of s relative to X, and X is the result of changes in S and a. Instance:

6. The gradient of the determinant

For the derivation of the determinant, the matrix A is nxn, we know that the determinant is related to the algebraic cofactor type,

is to remove the cofactor type after the J column of line I, then the derivative

Adj (a) with our linear algebra middle school is a meaning, so

"Cs229-lecture14" Singular value decomposition