Summary of linear discriminant analysis (LDA) algorithm

LDA is also called Fisher Linear discriminant, which is a non-iterative classification algorithm. The main idea of the algorithm is to project the sample in the feature space into a straight line in the space to realize the data compression from high dimension to one dimension. Obviously there are many orientations for such projections, and the LDA algorithm is looking for an optimal projection direction to minimize the distance between classes within the sample and the maximum distance between classes.

Projection direction: 1, after the projection of different categories of sample mean difference should be as large as possible, that is, the distance between classes as large as possible; 2. The sample variance of the same category should be as small as possible, i.e. the distance within the class should be as small as possible.

Define a Fisher criterion function (refer to the relevant textbook for how the benchmark function is determined), 650) this.width=650; "Src=" http://s3.51cto.com/wyfs02/M00/73/E5/ Wkiom1yjoungdeehaabydhjgwpm352.jpg "title=" Capture 7. PNG "alt=" Wkiom1yjoungdeehaabydhjgwpm352.jpg "/>

Where the vector w is the direction vector of the projection, the SB and SW are the first two classes and the intra-class dispersion matrix of the projection. In this way, the problem of the projection direction is transformed into the problem of seeking the maximum value of the criterion function. We can use the Lagrange multiplier method to solve the extremum of the function, so that the denominator of the function is a constant c as a constraint for the solution (because there is J (AW) =j (W) for any real number a. 650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/73/E2/wKioL1YJOw_Rlzn0AABHDmqb5Wo808.jpg "title=" Capture 8. PNG "alt=" Wkiol1yjow_rlzn0aabhdmqb5wo808.jpg "/>

Then it is Lagrange to the derivative, and finally get the best projection direction for:

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/73/E2/wKioL1YJQTnQcoJjAAAmduXtyck635.jpg "title=" Capture 9. PNG "alt=" Wkiol1yjqtnqcojjaaamduxtyck635.jpg "/>

If the Lagrange multiplier is not well understood see also: http://www.cnblogs.com/zhangchaoyang/articles/2726873.html

Algorithm steps:

1. Input training sample Set;

2. Calculates the mean vector of two categories;

3. Calculates the total dispersion matrix within a class;

4. The inverse array of the total dispersion matrix in the class is computed.

5. Get the projection direction, the formula is as above.
   

This article is from "Lu Yao" blog, please be sure to keep this source http://cwxfly.blog.51cto.com/6113982/1698935

Summary of linear discriminant analysis (LDA) algorithm