Singular value decomposition (SVD) of numerical analysis

[1] In many linear algebra problems, if we first think about what would happen if we did the SVD, then the problem might be better understood.

--lloyd n. Trefethen & David Bau, lll

In order to discuss the convenience of the problem and most of the problems encountered in practice, here we confine ourselves to the discussion of the real matrix, noting that the conclusions involved are also easy to extend into complex matrices (in fact, many textbooks use the description of complex matrices), in addition, the use of symbol x, y and other representations of vectors, A,b,q such as the expression matrix.

First, the concept of orthogonal matrix is given. The so-called orthogonal matrix , that is, the matrix of the different two-column vectors of the intercropping product equals 0 (the vertical definition in plane geometry is generalized in multidimensional cases), the same column vector and its own inner product equals 1 (unit vector). In particular, if Q is an n-order orthogonal phalanx, then Q ' q=qq ' =i, that is, the transpose of the orthogonal matrix is the inverse matrix of the matrix. The orthogonal matrix plays an important role in the numerical analysis, one of the main reasons is that it can keep the 2 norm of the vector constant (so SVD is also often used in solving the least squares problem), and the 2 norm of the Matrix and the F-norm invariant, i.e. | | Qx| | _2=| | x| | _2, | | aq| | _2=| | a| | _2,| | qa| | _f=| | a| | _f.

singular value decomposition theorem: for any one m*n real matrix A, there are m*m orthogonal matrix U and n*n orthogonal matrix V, and m*n diagonal matrix D=diag (d_1,d_2,..., d_r), making

A = UDV '

Among them,d_1>=d_2>=...>=d_r>=0 is called singular value , and the columns of U and V are called left singular vectors and right singular vectors respectively.

The geometrical meaning of SVD is given below, and then-dimensional unit vector x is a hyper-ellipsoid in M-dimensional space under the image of an arbitrary m*n matrix A=UDV ' . Specifically, the orthogonal transformation v ' keeps the vector length of x constant, the diagonal matrix D stretches the sphere to a super ellipsoid, and the final orthogonal transform U rotates the hyper-ellipsoid, but does not change its shape, see.

Figure 1 SvD geometry interpretation [2]

Matrix properties of singular value decomposition:

1. the rank of matrix A equals the number of non-0 singular values .

2. The range space of matrix A is equal to the space spanned by the front r column vector of U, while the 0 space of a is the space spanned by the n-r column vector behind V.

3. | | a| | _2=d_1, | | a| | _f=sqrt (D_1+...+d_r).

4. A's non-0 singular value squared equals the non-0 eigenvalues of AA ' and a ' a.

Note that once the singular value of a can be decomposed, according to the properties given above, the rank of a, the range of a or the base of 0 space, and the 2-norm of a, the f-norm, etc., are naturally able to be obtained. In this respect, SVD can be seen as a tool for solving these problems. In addition, it is widely used to solve least squares problem, regularization problem, low rank approximation problem, data compression problem, and classification problem in text processing [4] .

Careful children's shoes found that all the above conclusions are based on the SVD theorem is correct and can effectively calculate a given matrix a SVD decomposition. With regard to the first question, the mathematical induction can be used to prove [3]; the second problem, because the mathematical induction method is used in the proof, obviously it can not effectively solve the SVD decomposition of the concrete matrix, and the numerical solution SVD needs to use the symmetric matrix eigenvalue decomposition (a simple idea is to AA ' Eigenvalue decomposition , and then get a singular value decomposition, unfortunately, the method of numerical stability is poor , the details of the content does not unfold narrative).

Finally, the relation between the singular value of the matrix and the eigenvalues of the Matrix is explained. First, there is singular value decomposition for any matrix, and not all matrices have eigenvalue decomposition ; Secondly, the orthogonal matrix is used in singular value decomposition, and the base used in eigenvalue decomposition is not orthogonal; Thirdly, the minimum singular value of the matrix is less than the modulus length of the minimum eigenvalue of the Matrix, The maximum singular value of the matrix is greater than the modulus of the maximum singular value of the matrix [5].

Reference documents:

[1] Numerical linear algebra chap4-5,l N. Trefethen,david Bau, lll, Lu Jinfu, kanji translation, people's post and Telecommunications publishing house, 2006

[2] Applied numerical linear algebra, J W. Demmel, Wang Guorong, People's post and Telecommunications publishing house, 2007

[3] Matrix calculation (third edition), Gene H.golub,Charles F.van Loan, Shing, people's post and Telecommunications publishing house, 2011

[4] The beauty of mathematics Chap15, Wu, People's post and Telecommunications publishing house, 2013

[5] The eigenvalues of the matrix A and the singular value of the size of the relationship? https://www.zhihu.com/question/40181430/answer/85446211

Singular value decomposition (SVD) of numerical analysis