[Dimensionality Reduction] PCA Principal Component Analysis

In fact, should be the first to tidy up the PCA, Zennai has no time, may be their own time is not sure, OK, below into the topic.

the concept of dimensionality reduction

The so-called dimensionality reduction is to reduce the dimensionality of the data. In machine learning is particularly common, before doing a picture to extract the wavelet feature, for a picture of a size of 800*600, if each point to extract five scale, eight directions of the characteristics, then each pixel point extract 40 features, then a picture is 40*800*600= 19.2 million characteristics. That is, we use a 19200000 vector to describe a picture. If there is a M picture, then the characteristic is the size of the m*19200000. Obviously this dimension is too large, so we need to reduce the dimension.

Formal Description: Assume that the data matrix is xm,n (this matrix may be a feature matrix, or it may be another matrix). m represents the size of the data, n represents the dimension, usually n is very large. The goal of dimensionality reduction is that we want to find a matrix y=xp, so that y satisfies the following two conditions:

(a) The dimension of Y is less than the dimension of x (this is the purpose of dimensionality reduction).

(b) The greater the difference between the dimensions of Y, the better (the greater the difference, the more information it contains, the more effective it is to describe the data);

Conditions (a) are relatively easy to meet, we only need pn,t (t<m), then the resulting y is ym,t obviously has the purpose of dimensionality reduction. So our focus is to find P satisfaction (b).

To do this, we first give the criterion-variance and covariance of the opposite-sex metric.

Variance and covariance

For data Matrix Xm,n, the difference of the J-dimensional data can be expressed by variance:

For data xm,n, what we want is that the greater the difference between all its dimensions and dimensions, the better, the difference between dimension p and Q can be expressed by covariance:

Thus, the differences in the dimensions of the data xm,n can be represented by the covariance matrix C

Obviously, the covariance matrix is a real-to-match matrix. It is still cumbersome to calculate the covariance matrix from the above equation, we operate directly on the matrix and center the matrix X.

Then the covariance matrix can be obtained by the following formula

It is quite simple to calculate the covariance matrix.

With the concept of the covariance matrix, we proceed to the previous discussion: As mentioned earlier, we need a Y=XP (centralized), the greater the difference between its dimensions, the smaller the covariance of Y covariance matrix D, the less the better, the formal expression is

This step is quite clear. Remember what kind of D we need, this d better be a diagonal array, so that the elements that are not on the diagonal are all 0. The problem turns into the diagonalization of real symmetric matrices. According to the knowledge of matrix theory, we know that a symmetric matrix must exist an orthogonal matrix p to make the Ptap a diagonal array.

In principle, even if this is finished, but this force has a limit, a is linearly independent (or a has n different eigenvalues), at this point a of the eigenvalues from the big to the small sort, each characteristic value corresponding to a eigenvector, we put these eigenvectors from left to right in the row up to get p. If we take the first T eigenvalues, then the size of P is n*t, and by Y=XP, we get the Y is a m*t matrix (x is the m*n matrix), which plays a role in dimensionality reduction. Of course, if the size of P is n*n, then there is no goal of dimensionality reduction, but the x is mapped to a new space.

From the geometrical point of view, in fact, the linear transformation is a spatial mapping, we do not change the location of the data in space, but with a different radicals to represent him, about the base feeling this blog is in place:

Http://blog.codinglabs.org/articles/pca-tutorial.html

give a simple matlab code

m=10n=100;t=50; x=100*rand ([M,n]), percent original datax=x-repmat (mean (x), m,1),%centralizationc= (x' *x)/m; [e,d]== sort (diag (-= E (:, order);%=D (:, order); Y=x*e (:, 1:t); %Y

[Dimensionality Reduction] PCA Principal Component Analysis