Eigenvalue decomposition, singular value decomposition (SVD)

Eigenvalue decomposition and singular value decomposition are all visible methods in the field of machine learning. There is a close relationship between the two, and I'll talk about both eigenvalue decomposition and singular value decomposition for the same purpose, which is to extract the most important features of a matrix.

1. Characteristic values:

If a vector v is a eigenvector of square A, it will certainly be represented in the following form:

Written in matrix form:

At this time λ is called the eigenvalues of the Eigenvector V, and a set of eigenvectors of a matrix is a set of orthogonal vectors .

2. Feature decomposition:

Eigenvalue decomposition is the decomposition of a matrix into the following form:

where q is the matrix of the eigenvectors of this matrix A, the orthogonal matrix is reversible. Σ?=?diag (λ1,?λ2,?...,? λn) is a diagonal array, and the element on each diagonal is a characteristic value.

First of all, to be clear, a matrix is actually a linear transformation, because a matrix is multiplied by a vector, which is actually equivalent to a linear transformation of the vector.

When the matrix is a high-dimensional case, then the matrix is a linear transformation in the high-dimensional space, this linear change may not be represented by the image, but it can be imagined that the transformation also has a lot of direction of transformation, we are decomposed by the eigenvalue of the first n eigenvectors, Then it corresponds to the main n change direction of the matrix. we can approximate this matrix (transformation) by using the first n direction of change. That's what we said before: extracting the most important features of the matrix. summing up, eigenvalue decomposition can get eigenvalues and eigenvectors, the eigenvalues represent how important this feature is, and eigenvectors represent what this feature is , and each feature vector can be understood as a linear subspace, We can use these linear subspace to do a lot of things.

However, eigenvalue decomposition also has a lot of limitations, such as the transformation of the matrix must be a square .

3. Singular value decomposition

Eigenvalue decomposition is a good way to extract matrix features, but it only applies to square matrices . In the real world, we see most of the matrix is not a square, for example, there are M students, each student has n subjects, so that the formation of a M * n matrix may not be a square, how can we describe such a general matrix as the description of the characteristics of the important characteristics of it? Singular value decomposition is used to do this, singular value decomposition is a decomposition method that can be applied to arbitrary matrices.

singular value decomposition is a decomposition method that can be applied to arbitrary matrices :

Suppose A is a matrix of M * N, then the resulting U is a square of M * M (the vector inside is orthogonal, the vector inside U is called the left singular vector ),σ is a real diagonal matrix of M * n (the elements outside the diagonal are all 0, The elements on the diagonal are called singular values , andVT (transpose of V) is a matrix of n * n , the vector inside is also orthogonal, the vector inside V is called the right singular vector . From the picture to reflect the size of several multiplying matrices can get the picture below

So how do singular values and eigenvalues correspond? First, we will transpose a matrix A at * A, we will get the ATA is a square, we use this square to find the eigenvalues can be obtained: Here the V is the right singular vector above us. In addition we can also get:

The ōi here is the singular value mentioned above, the UI is the left singular vector mentioned above.

It is common practice to arrange singular values from large to small. So σ can be determined by the M single.

Singular value σ is similar to the eigenvalues, in the matrix σ is also from the large to the small arrangement, and σ reduction is particularly fast, in many cases, the first 10% or even 1% of the singular value of the sum of the total singular value of more than 99% . In other words, we can also approximate the matrix with the singular value of the former R large, which defines the partial singular value decomposition :

R is a number that is much smaller than M, N, so that the multiplication of the matrix looks like this:

The result of multiplying the three matrices on the right will be a matrix close to a, where R is closer toN, and the result of multiplying is closer to a. And the area of the three matrices (in the storage point of view, the smaller the size of the matrix, the less storage) is much smaller than the original matrix A, if we want to compress space to represent the original matrix A, we save the three matrix here: U, Σ, V is good.

Reference:

Mathematics in Machine learning (5)-powerful matrix singular value decomposition (SVD) and its application

Eigenvalue decomposition, singular value decomposition (SVD)