dimensionality reduction of machine learning algorithms

In the process of machine learning, we often meet the problem of fitting. The high dimension of input data or features is one of the problems that lead to overfitting. The higher the dimension, the more sparse your data will be in each feature dimension, which is basically catastrophic for machine learning algorithms. There are many ways to reduce dimensions. What we are going to talk about today is the LDA dimensionality reduction.

The idea of Lda dimensionality is: If two types of data are linearly separable, i.e. there is a hyper-plane, separating the two types of data. Then: There is a model rotation vector, which projects two kinds of data onto one dimension and is still linearly divided.

Ask questions

Suppose a set of n tagged data (xi,ci) is undefined, where the tag C is divided into two categories, namely Ci = 0 or Ci=1, and the classifier is designed to separate the data. If the dimensions of X are high, even more than N, then dimensionality is required.

Problem solving Process

1, according to the linear transformation, the X is reduced to one-dimensional

Assuming that the rotation vector is W, the data x is projected to one-dimensional y, resulting in Y =wtx, where input data x, rotation vector W.

So the original x-dimensional vector into one-dimensional, using the classification algorithm to classify the data into C. Thus, threshold values can be found W0, if Y>w0 is a class, Y<w0 is a class.

2. Calculate the class mean and variance of each category

So that the C1 class has N1 elements, C2 has N2 elements, calculates the mean and the mean and loose (variance) in the class before the projection:

3. Find Fisher criteria

4, optimization of the objective function

Which is to take the extremum after the derivation of the objective function.

The countdown is:

The derivation is obtained, the three are in the same direction.

Principal component Analysis PCA------Subject Model

The difference between PCA and LDA

LDA: The best way to classify performance

PCA: The direction of the sample point projection with the maximum variance

Practical problems often need to study several characteristics, and these characteristics have a certain relevance.

Combine multiple features into a few representative features. The combined features can represent most of the information of the original features, and they are not related to each other and reduce the correlation. This method of extracting the principal component of the original feature is called principal component analysis.

The question is raised:

For the data of M samples containing n features, each sample is labeled as a row vector, and the matrix A is obtained:

The idea of solving problems:

Look for the main direction of the sample U: Project The values of the M samples onto a line l to get the points of M located on the line L, and calculate the variance of the m projection points. The line direction with the largest variance is considered to be the main direction.

Suppose the sample goes to mean value.

The core derivation process of the PCA in the variance calculation

Calculate the value of the Axu by taking the extension direction U of the projected line L

Find the variance of vector a X u

Objective function: J (u) = Utatau

The objective function is to seek the stationed point:

Since the U-number multiplication gets the same direction as u, increasing u is the constraint of the unit vector, i.e.: | | u| | 2=1 = UTu

To establish the Lagrange equation:

L (U) = Utatau-λ (uTu-1)

Derivation:

Analysis Atau =λu

If the samples in a are all de-n-1, then the covariance matrix of ATA and A is only the difference coefficient.

U is the characteristic vector of the ATA, and the value of λ is the variance of the projected value of the eigenvector of the original observed data in the direction of the vector u

Important applications of PCA

Denoising, dimensionality reduction, pattern recognition, analysis of data Hu correlation and multi-source fusion

dimensionality reduction of machine learning algorithms