Another talk about PCA

In fact, previously written in the PCA-related blog post, but because the theoretical knowledge is limited, so the understanding is relatively shallow. This blog post, we understand the PCA in a different way, I assume that we have a preliminary understanding of PCA. First, let's cite an example in a two-dimensional space, such as:

The left image represents five points in a two-dimensional space, and we try to find a projection direction so that the variance of these 5 points is the largest in this direction. Here are two examples of the figure and the right, it is clear that the variance of the graph is the largest projection direction of the difference in the direction of the projection. PS: here to explain the meaning of variance, purely personal understanding. Imagine that if we had a projection direction A and B, the original feature was almost centered at a point after a projection, meaning that the variance was almost zero. At this point we can assume that the characteristics of the projected sample are almost the same on this dimension. As a result, we have very little information to reflect in the new space, and we have reason to think that the projection direction is very poor. Conversely, if the sample has a very large variance in the new feature space after the B-projection, we think the information on this dimension of the new sample is very clear, meaning that the projection direction is a "main projection direction". The purpose of PCA is to find the best K "main projection direction", how to judge the direction of the projection is good, we have said, with the variance: the greater the difference between the projection behind the better.

Next we use the mathematical formula to prove that: suppose we have a M sample, U is to solve the projection direction, for the sample point, here has been to the mean value, then the maximum variance can formalize the following formula:

PS: As explained here, this is the variance of the formula, the sample after the projection of the corresponding new sample is, because the original space to mean that the new sample is also going to mean, so the variance of the new sample can be expressed as, the following formula does not explain, and here.

Thus, we turn the problem of maximizing variance into the problem of solving the eigenvectors of this equation. As for why this is so, mathematically proven, as follows: (the city Lagrange equation used here to solve the maximization problem)

You can get a derivative of u:

The derivative is zero to know that you are the eigenvector, and these eigenvectors are orthogonal, so we select the maximum k eigenvalues corresponding to the eigenvector can constitute the largest variance subspace, sample projection in this subspace each dimension of the maximum variance.

Summary: Although the blog is short, but for some of the PCA has a certain understanding of the people do not know why, I believe it is worth a look.

Another talk about PCA