Simple yet graceful SVD decomposition

I. Introduction--high-dimensional space and watermelon

This semester has a "network data mining", the original is particularly concerned with this semester selected a "Pattern recognition and data mining" in a certain degree of repetition, but fortunately, the teacher is not scripted, speaking more personal understanding and experience.

What's interesting today is that when it comes to clustering, the teacher is somewhat dismissive, saying that in high-dimensional spaces, clustering algorithms may not be as ideal as we think, such as when we use the K-means algorithm, we generally think that the distribution of data (in two dimensions) is like this:

But often the situation may not be as intuitive as we see it, for example if we assume that the data is distributed in a one-dimensional space, and that the sample distribution is about half the total range of values:

So we intuitively have:

$${R_{SAMPLE1D}} = {1 \over 2}$$

Further, if we are in two-dimensional space, in each dimension, the sample value interval is also half the total value interval, then the sample distribution space occupies the total space size:

$${r_{sample2d}} = {1 \over 4}$$

So can continue to recursion, imagine, if you think in the high-dimensional space sample is a super-sphere (like a watermelon), then some out of the field, in our less care of the rind part, is occupied the vast majority of space.

In front of these, is a primer, our understanding of the high-dimensional space is much more insensitive than we think, and it is very ironic that we are faced with the problem is often high-dimensional data, our approach is often only by virtue of mathematical tools to explore, when any subjective assumptions tend to deviate from the actual.

Two. SVD Decomposition introduction

1. Introduction to the Model

Start talking about SVD decomposition.

In fact, SVD decomposition is not a probabilistic model, even with the non-probabilistic model we encountered in the ordinary.

Usually we encounter the SVD decomposition, often to find a discriminant surface, such as perceptual machines, SVM, etc. (decision tree is a plurality of classification surface)

The graceful nature of SVD decomposition is that the complex clustering analysis is accomplished simply by the substitution of pure mathematics, and provides the numerical basis without any ambiguity.

Simply put, SVD decomposition, is the original matrix a split into three matrix product, these three matrices are U, Σ, V

Also that

$ $A = u\sigma {v^t}$$

Where v describes the orthogonal base in the primitive space, U describes the orthogonal basis of the relevant space, and Σ describes a multiple of the stretch when the vector in V becomes a vector in U.

2. Model Application

For the application of SVD decomposition, there are a lot, a brief mention, but do not expand to the point.

(1) Cluster division (recommended system, text mining ...) ）

The application scenario is to consider the original matrix D as a probability distribution matrix (but not necessarily normalized)

Then obviously, the entire matrix decomposition process can be written for this conditional probability of a decomposition, specifically can refer to Su Jianlin's blog why SVD means clustering.

(2) Data de-noising

That is, the noise is set to 0 according to the energy size, thereby filtering out the noise of the data.

(3) Data compression

SVD decomposition is originally a large matrix is disassembled into a few small matrices, this compression especially in the data sparsity and low rank when the expression of more fully.

Three. The mathematical derivation of SVD decomposition

SVD decomposition is only a mathematical transformation.

We know that a matrix (square) transformation is equivalent to a linear spatial transformation.

Specifically:

(1) A matrix consisting of an orthogonal base corresponds to this rotational stretching transformation

(2) A diagonal matrix to a stretch transform

(3) a matrix (orthogonal matrix) corresponding to a unit orthogonal basis corresponds to a rotation transformation

Its derivation can be referenced from the following two blogs

(1) Eigenvalue decomposition, singular value decomposition (2) explanation and derivation of singular value decomposition (SVD) principle

The above two blogs are from the eigenvalue decomposition of the ordinary square to the singular value decomposition of the matrix, and the geometric meaning of singular value decomposition is introduced.

Four. The meaning of SVD decomposition

We might ask a question, why does SVD mean clustering?

We can look at this:

(1) SVD decomposition is the decomposition of a matrix into its left feature space matrix and the right feature space matrix, and the middle of the eigenvalues matrix is a representation of the weights, so we each time as long as the selection of those important features to survive, we can almost no loss of the original matrix reduction.

(2) We can see that for each cluster of the left characteristic matrix and the clustering of each right feature matrix is one by one corresponding, this is also reasonable, because in the original matrix they are interdependent, at the same time, the UV matrix is an expression of the original matrix, they have a one by one mapping relationship.

(3) SVD decomposition can be regarded as a low-rank expression of the original matrix, that is, because the original matrix is necessarily a certain coupling between the project, so the original matrix, whether from the line or from the column, there must be a lot of linear correlation (or approximate) relationship, which is a very important premise of clustering.

Simple yet graceful SVD decomposition