pca--Principal Component Analysis

PCA (Principle Component analysis) is mainly used to reduce the dimensionality of the data. For high-dimensional data, processing is cumbersome, and high-dimensional data may contain related dimensions, data redundancy, PCA by the high-dimensional data to the low-dimensional mapping, as far as possible to preserve the information contained in the data, to achieve the purpose of simplifying the data.

Assuming that the original data is represented as $\{{{x}_{1}},{{x}_{2}},\cdots, {{x}_{n}}\}$ has $n$ data, ${{x}_{i}}$ is $d$, and now first analyzes how PCA maps it to one dimension and then to multidimensional.

In order to map data to a one-dimensional mapping, there are two problems that need to be solved, one is to map to that direction, but to determine how to map after the direction. Let's go backwards and think about the second question, and then the first one.

In addition, before using PCA, it is common to normalized the different dimensions of the data to eliminate the effects of different dimension scales.

1, the Data ${{x}_{i}}$ transformation ${{x}_{i}}^{\prime}={{x}_{i}}-m$, wherein the $m$ is the sample mean value. This step makes the data average value $\vec{0}$;

2, further transform ${{x}_{i}}^{\prime \prime}={{{x}_{ij}}^{\prime}}/{{{\sigma}_{j}}}\;$ where ${{\sigma}_{j}}$ is the standard deviation of the $j$ dimension of the data, This step makes the standard deviation of each dimension of the data to be 1;

After the above two steps, the effect of the different dimension of the sample PCA is basically eliminated. Why eliminate the effects of different dimensions can be understood in terms of sample variance. (a dimension with a lower variance is "masked" by a dimension with a higher variance, but this does not mean that the former contains less information than the latter).

all of the following discussions will assume that the data has been normalized, and the mean value is $m=\vec{0}$ , each dimension variance is 1 . without normalization, the derivation is not affected, but the derivation process is a little more complicated.

First, Suppose the mapping direction is given, how to map?

To one-dimensional mapping is to map the original $d$ dimension data to a straight line, now assume that the direction of the line is given, consider the problem of mapping, intuitive, we first think of the original data in this direction to do vertical projection, the following will show that our intuitive feeling is very reasonable.

The above two charts are two different mappings, the first is the vertical projection, the second is very wonderful, because I painted, but it is also a kind of mapping good ~ Which way is better? This will take into account the purpose of PCA. The PCA function simplifies the data by dimensionality reduction, but this simplification is undesirable if the simplified data cannot result in subsequent analysis with the approximate results of the original data. Therefore, the mapping in the dimensionality to preserve as much as possible the information contained in the data, that is, the mapped data should be as close as possible to the original data, so the minimum error can be used as a mapping criterion.

The aim is to make the error function minimal. This is obviously about the convex function of the ${{\alpha}_{i}}$, so the partial derivative of the ${{\alpha}_{i}}$, which is 0, can get the solution \[\begin{align}{{\alpha}_{i}}={{e}^{t}}{{x}_{i}}\ End{align}\]

What does that mean? This means that the mapped data point is exactly the vertical projection of the original data point toward the direction $e$. This solves the problem of how projections are projected after a given direction-vertical projection, after projection, the original data ${{x}_{i}}$ represented as one-dimensional data ${{\alpha}_{i}}$.

Two, which direction to project?

This solves the problem of how to map data in a given direction $e$, and now discusses how to determine $e$ from two angles.

Angle 1 : Minimum error criterion

Note $s=\sum\limits_{i=1}^{n}{({{x}_{i}}-m) {{{x}_{i}}-m)}^{t}}}=\sum\limits_{i=1}^{n}{{{x}_{i}}x_{i}^{t}}$ (Assuming that the sample mean is $\vec{0}$), called the sample scatter matrix, it is a constant than the covariance matrix of the sample, representing the dispersion of the sample. The dispersion matrix is brought into (4) to get the error function \[\begin{align}j (e) =-{{e}^{t}}se+\sum\limits_{i=1}^{n}{x_{i}^{t}{{x}_{i}}}\end{align}\]

Minimization of $j (e) $ is equivalent to minimizing $-{{e}^{t}}se$, so further simplification (5) is \[\begin{align}j (e) =-{{e}^{t}}se\end{align}\]

In addition, since we stipulate that $e$ is a unit vector, minimizing (6) is an optimization problem with equality constrained ${{e}^{t}}e=1$, and writes out the corresponding Lagrangian function \[\begin{align}j (E,\LAMBDA) =-{{e}^{t}}se+\ Lambda {{e}^{t}}e\notag\end{align}\]

To $e$ the partial derivative and make it 0, get \[\begin{align}se=\lambda e\end{align}\]

The $e$ of the required solution is exactly the eigenvector of the sample scatter matrix, and $\lambda $ is the corresponding characteristic value. Will (7) return (6) Get \[\begin{align}j (e) =-{{e}^{t}}se=-\lambda\notag\end{align}\]

So if the error function is minimal, $\LAMBDA $ should be the maximum eigenvalue, $e $ should be the eigenvector corresponding to the maximum eigenvalues of the scatter matrix $s$.

Angle 2 : Maximum Variance

Now take a different angle to consider the variance of the data after the map is completed, the larger the variance, the better, because the projection mode is given (vertical projection), then the larger the data variance after projection, it is shown that the data after projection reflects the original data distribution characteristics. After the original data ${{x}_{i}}$ projection, the corresponding data is ${{\alpha}_{i}}={{e}^{t}}{{x}_{i}}$, then the variance of the projected data is \[\begin{align}d (e) &=\sum\limits_{i =1}^{n}{\alpha _{i}^{2}}\notag \\& =\sum\limits_{i=1}^{n}{{{e}^{t}}{{x}_{i}}x_{i}^{t}e}\notag \\& ={{e}^{T} }se \notag\\\end{align}\]

The visible maximized $d (e) $ and minimized (5) (6) are consistent.

Alternatively, you can add (5) $j (e) $ and $d (e) $ to see that the result is $j (e) =-{{e}^{t}}se+\sum\limits_{i=1}^{n}{x_{i}^{t}{{x}_{i}}}$, which is the variance of the original data (assuming all data mean value is $ \vec{0}$). While $d (e) $ is the variance of the mapped data, $J (e) $ can be understood as the loss of the data variance due to the mapping process, which also illustrates the equivalence of maximizing $d (e) $ and minimizing $j (e) $, in a word, the mapping maximizes the variance of the data, the variance is the change, Changes reflect the information of the data.

iii. Summary

Through the above two steps, the original data to one-dimensional data mapping process is complete. First, calculate the maximum eigenvalue of the scatter matrix $s$ for the normalized eigenvector $e$, and then calculate the projection result ${{\alpha}_{i}}={{e}^{t}}{{x}_{i}}$.

Obviously, only one-dimensional mapping can be a lot of data loss information, depending on the distribution of the data itself. In general, it is more than a few dimensions to map, the original data mapped to 3-dimensional, 4-dimensional and so on. Mapping to a few dimensions without standards, the higher the mapping dimension, the more you can retain the original image of the data, but also less than the purpose of reducing dimensionality to simplify the data, so need a trade-off between the two ~

How do you map to a higher dimension? And one-dimensional analogy to know, if you want to map to $p$ dimension, this $p$ direction should be the maximum $p$ of the walk matrix eigenvalues corresponding to the direction of the eigenvector.

\[x=\left[\begin{matrix}x_{1}^{t} \\x_{2}^{t} \\\vdots \\x_{n}^{t} \\\end{matrix} \right]\], the normalized feature corresponding to the maximum eigenvalues of a walking matrix $p$ Vector is $e=\left[\begin{matrix}{{e}_{1}}, & {{e}_{2}}, & \cdots, & {{e}_{p}} \\\end{matrix} \right]$, then the PCA process can be written For \[y=xe\]

A data point that is mapped after each behavior of the $Y $.

In addition, $Y $ can be represented as \[\begin{align}y& =\left[{{y}_{1}},{{y}_{2}},\cdots, {{y}_{n}} \right] \notag\\& =[X{{e}_{1}},X{{e}_ {2}},\cdots, X{{e}_{n}}] \notag\\\end{align}\]

so \[{{y}_{i}}=x{{e}_{i}}=x{{e}_{i1}}+x{{e}_{i2}}+\cdots +x{{e}_{in}}\]

${{y}_{i}}$ is the $i$ principal component, which is a linear combination of the original data.

The eigenvector of the scatter matrix embodies the direction of the change of the data, the more the characteristic vectors of the eigenvalues reflect the main direction of the data, so the original data information contained in each dimension of the map is decremented, and the main information of the original data is embodied in the previous dimensions of the map. This is the reason why the mapping process chooses the mapping direction according to the eigenvalue size in turn. There are no strict gold standards for $p$ choices, but there are some rules of thumb, such as the Kaiser-harris rule, which suggests selecting a eigenvector that corresponds to a eigenvalue of more than 1.

pca--principal Component Analysis