Non-common dimensionality reduction method: Laplacian eigenmaps Laplace feature mapping

Original Address Laplace Matrix definition of Laplacian matrix

Talking about the dimensionality reduction technology in machine learning, probably most of the friends who know a little about machine learning know PCA, and today we introduce a new dimensionality reduction Method-Laplace feature mapping

The Laplace matrix (Laplacian matrix), also known as the Kirchhoff Matrix, is a matrix that represents a graph. Given a graph g= (v,e) with n vertices, its Laplace matrix is defined as: l=d-w

where d is the degree matrix of the graph and W is the adjacency matrix of the graph. (Do not know the degree matrix and the adjacency matrix of the self-Baidu) Laplace Matrix L is a symmetric semi-positive matrix, L 1 = 0 1, that is, the minimum eigenvalue is 0, the corresponding eigenvector is. Proof: l* 1 = (d-w) * 1 = 0 = 0 * 1. L have n non-negative real eigenvalues and for any one of the real vector F, the following formula is established:

The proof is as follows:
Laplacian eigenmaps Laplace feature mapping

Laplacian Eigenmaps is to construct the relationship between data with a local perspective. If two data instances I and J are similar, then I and J should be as close as possible in the target subspace after dimensionality reduction. Its intuitive idea is to hope that the points that are related to each other (the points connected in the graph) are as close as possible in the space after dimensionality reduction. Laplacian Eigenmaps can reflect the internal manifold structure of the data.

The specific steps for using the algorithm are:

Step 1: Build the diagram

Use a method to build all the points into a graph, for example, using the KNN algorithm, to connect the nearest K points of each point to the top. K is a pre-set value. The graph matrix that is constructed is a sparse matrix that retains only the most similar K-neighbor relationships.

Step 2: Determine weights

Determine the weight between points and points, for example, using the Thermonuclear function to determine (of course, this place you can choose other similarity measures to measure), if the point I and Point J are connected, then the weight of their relationship is set to:

The feature vectors corresponding to the minimum m non-0 eigenvalues are used as the output of the reduced dimension results.

As mentioned earlier, Laplacian Eigenmap has the characteristic of distinguishing data points, which can be seen from the following examples:

As shown in Figure 1, the graph on the left shows two types of data points (data is a picture), the middle diagram shows the position of each data point in two-dimensional space after Laplacian Eigenmap, the right figure represents the result of using PCA and taking the first two main direction projection, can clearly see, on this classification problem , the results of Laplacian Eigenmap were obviously better than that of PCA.