PCA and Lle

Reference: Lle principle Summary

Personal understanding

the defect of PCA dimensionality reduction : There are some linear relationships between samples in high dimensional space, and these relationships are not preserved after dimensionality reduction. For example, in a high-dimensional space, the shortest path is not the shortest line between two points in three-dimensional space, but the distance of the surface, and when we drop the dimension, it becomes the projection distance.

Therefore, we want to be able to preserve the distance of the original space (just an example), such as expanding the original surface rather than projecting it directly into the plane.

LLE algorithm :

Unlike PCA, LLE preserves local linear relationships in high-dimensional spaces.
Before lle, the linear relationship between each sample and its nearest K sample is--weights, and the linear relationship is applied to each sample after the dimensionality reduction space.

Add

Reference: PCA and kernel PCA

"About dimensionality reduction/redundancy"

How to understand redundancy?

such as two-dimensional plane, y=x near the data point has a strong linear correlation, that is, one of the dimensions is redundant, as long as we know one of the dimensions, we can roll out another dimension, so you can reduce the dimension, for this example can be translated + rotated, so that it is distributed on the x-axis.

How to use mathematics to express this idea?

The variance/covariance is maximized, that is, the maximum variance between the projected points. The relationship between variance and covariance: When a number of points are normalized and centralized, the maximization of the two methods is equivalent, and the maximum of covariance is generally directly obtained. The formula derivation for the specific reference link.

"Kernel PCA"

Suppose K is known.

As with the above, we also require the maximization of covariance ,
\[\begin{align}c&=\sum_{i=1}^{n}\phi (x_i) \phi (x_i^t) \ &=[\phi (x_1),..., \phi (x_n)]\begin{bmatrix}\phi (x_ 1) ^t\\...\\\phi (x_n) ^t\end{bmatrix}\ &=x^tx\\end{align}\]

However, φ is unknown or difficult to calculate, so we try to solve it by means of a kernel function.
\[\begin{align}k&=xx^t\&=\begin{bmatrix}\phi (x_1) ^t\\...\\\phi (x_n) ^t\end{bmatrix}[\phi (x_1),..., \ Phi (X_n)]\&=\begin{bmatrix}\phi (x_1) ^t\phi (x_1) &, ... &\phi (x_1) ^t\phi (x_n) \...&...&...\\phi ( X_n) ^t\phi (x_1) &...&\phi (x_n) ^t\phi (x_n) \end{bmatrix}\&=\begin{bmatrix}k (x_1,x_1) & .... &K (x_ 1,x_n) \...&...&...\k (x_n,x_1) &...&k (x_n,x_n) \end{bmatrix}\end{align}\]
"Note" Here the k=xx^t and the required covariance x^tx are not equal, but there must be some relationship between the two:
\[\begin{align}xx^tu&=\lambda u&u is a feature vector \x^tx (x^tu) &=\lambda (X^TU) &x^tu as a eigenvector, but not necessarily a unit \ \ End{align}\]
Therefore, to x^tu the feature vectors:
\[v=\frac{x^tu}{| | x^tu| |} =\frac{x^tu}{\sqrt{u^txx^tu}}=\frac{x^tu}{\sqrt{u^t\lambda u}}=\frac{x^tu}{\sqrt{\lambda}}\ wherein, u^tu=1,v can be regarded as a directional axis /dimension \ \alpha=\frac{u}{\sqrt{\lambda}}, for a column vector V, so: v=\sum_{i=1}^{n}\alpha_i\phi (x_i) \]
However, X^t is still unknown, so V is also unknown, that is, the directional axis of the feature space of the high dimension is unknown, but we can directly seek the projection of φ (XJ) in the V direction of the feature space (this is our ultimate goal):
\[\begin{align}v^t\phi (X_j) &=\frac{u^tx\phi (X_j)}{\sqrt{\lambda}}\&=\frac{u^t}{\sqrt{\lambda}}\ Begin{bmatrix}\phi (x_1) ^t\\...\\\phi (x_n) ^t\end{bmatrix}\phi (X_j) \&=\frac{u^t}{\sqrt{\lambda}}\begin{ Bmatrix}k (X_1,x_j) \\...\\k (x_n,x_j) \end{bmatrix}\end{align}\]

Therefore, we can obtain the projection of the high dimensional space by only asking the eigenvalues of the kernel function and the corresponding unit eigenvector.

PCA and Lle