Matrix Singular Value Decomposition and Its Application

This paper is a assignment assigned by the teacher in the matrix analysis class. The main content is from the http://www.ams.org/samplings/feature-column/fcarc-svd article. In this case, the full text is not translated by a single word. Some content is added and some content is deleted.

I. Basic Knowledge

1. feature value:

If we say a vectorVIs the feature vector of phalanx A, which can be expressed in the following form:

In this case, λ is called a feature vector.VCorresponding feature value. A group of feature vectors in a matrix is a group of orthogonal vectors. Feature value decomposition refers to dividing a matrix into the following forms;

Q is a matrix composed of feature vectors of this matrix A. Σ is a diagonal matrix, and each element on the diagonal is a feature value.

2. Singular Values:

Suppose M is a matrix of mxn, and all the elements belong to the domain K, that is, the real or complex number. In this case, there is a decomposition mechanism, where U is the mxm-order matrix, and Σ is the semi-definite mxn-order diagonal matrix, while V * is the derivative transpose of V, which is the nxn-order matrix. Such decomposition is called the Singular Value Decomposition of M, and the element Sigma IJ on the Σ diagonal is the singular value of M.

2. geometric representation of linear transformation and Singular Value Decomposition

First, let's look at a simple matrix, such as the matrix of the two rows and two columns below:

You need to know that a matrix is actually a linear transformation, so the above matrix m actually corresponds to the following linear transformation:

Because this matrix m is multiplied by a vertex.(X, y)The result is:

 

This means that the X coordinate is stretched horizontally.

It is easy to know that the matrix above is symmetric, so this transformation is an extension transformation to the horizontal direction of the X axis. When the matrix is not symmetric, for example, the following matrix:

It describes the transformation as follows:

We can see that this transformation is not horizontal stretch, but to the upper right corner of the 45 ° angle.

The above two examples involve a single matrix. Next, let's take a look at the geometric representation of Singular Value Decomposition, the essence of Singular Value Decomposition is the transformation process from an orthogonal grid to another orthogonal grid (Note: Only the European space is taken into account, and the single space is not considered ).

Consider a set of orthogonal unit vectorsV1, V2And another set of orthogonal vectors mV1,MV2Such:

 

We useU1AndU2As MV1, MV2Vector of units in the direction, and m is represented by σ 1, σ 2V1, MV2These values are called the Singular Values of M. We can get

MV1= σ 1U1

MV2= σ 2U2

For a vectorXIn the first coordinate system, we can write

This means that

Because

MXCan be changed

Usually recorded

Where u isU1AndU2A matrix composed of two column vectors. Σ is a diagonal matrix composed of σ 1, σ 2, and V is composedV1, V2A matrix composed of two column vectors.

Iii. Application-Data Compression

Singular Value Decomposition can be used to more effectively represent data. Assume that we want to convert the following image containing 15x25 black and white pixels:

Considering that only the three types of columns shown below are shown in the figure, the data can be stored in a compact manner.

We can use a 15x25 matrix to describe this image. If the element at a position is 0, it indicates a black spot, and 1 indicates a white spot. In this way, a matrix containing 375 elements is constructed.

If we perform Singular Value Decomposition on m, we will find that there are only three non-zero Singular Values.

σ 1 = 14.72

σ 2 = 5.22

σ 3 = 3.31

Therefore, the matrix can be expressed

This also means that we have three vectors each containing 15 elements.,At the same time, there are three vectors with 25 elements each and three singular values. Therefore, we can use 123 elements instead of 375 elements to represent this matrix. In this way, Singular Value Decomposition discovers redundant information in the matrix and provides a way to remove redundant information.

Iv. Application 2-Noise Reduction

The previous example shows how to deal with a matrix where most singular values are 0. Generally, large Singular Values are information that we are interested in. For example, suppose we use a scanner to scan the image below into a computer, but there are some flaws in the image during the scanning process (usually referred to as "noise ")

We use the same processing method: Use a 15x25 matrix to represent data and perform Singular Value Decomposition. The obtained singular values are as follows:

σ 1 = 14.15

σ 2 = 4.67

σ 3 = 3.00

...

σ 15 = 0.05

Obviously, the first three singular values are the most important, so we assume that the other Singular Values are generated due to the noise in the figure, and we can make an approximation.

This will prompt the image quality

V. Application 3-Data Analysis

When we collect data, we often find noise. No matter how good our devices are, there will always be errors in the measured values. If we remember that the large singular values correspond to the important features in the matrix, it is a matter of course to use Singular Value Decomposition to learn and process the data just collected.

Here is an example. Suppose the data we have collected is as follows:

We can take these data points and put them into the matrix.

-1.03	0.74	-0.02	0.51	-1.31	0.99	0.69	-0.12	-0.72	1.11
-2.23	1.61	-0.02	0.88	-2.39	2.02	1.62	-0.35	-1.67	2.46

After Singular Value Decomposition, we can obtain the following two singular values:

σ 1 = 6.04

σ 2 = 0.22

We can see that the first singular value is far greater than the second one. We can rest assured that the small σ 2 is generated by the noise in the data, and the singular value should be 0. in this case, the rank of the matrix is 1, which means that all data is located inUIDefine online.

Vi. Summary

Due to the good mathematical characteristics of Singular Values, Singular Value Decomposition is not only applied to principal component analysis, image compression, digital watermarking, and document classification, it also has good applications in signal decomposition, Signal Reconstruction, signal noise reduction, Data Fusion, target recognition, target tracking, fault detection, and neural networks. In a word, Singular Value Decomposition provides an extremely effective means to apply the viewpoint of linear algebra to practical use.

 

PS: After completing matrix analysis this morning, it also brought a full stop to the last pure mathematics class in my learning career. Although teacher Wei Feng told us not to tell anyone to learn matrix analysis after going out (because the lesson is too short to talk about), this course taught me a lot of knowledge, from this article, we can see the wide application of SVD, which is just a typical highlight of matrix analysis. I remember when I was in my sophomore year, Beihang's Daniel told me that linear algebra was widely used in data mining. After reading the book "The beauty of mathematics" written by Wu Jun, I found that this is true. Obviously, mathematics is definitely the cornerstone of the computer it field. I believe I will see more in the future and more beautiful in mathematics.