Patterns Recognition (Pattern recognition) Learning Notes (vii)--linear classifier and linear discriminant function

1. Why design a classifier?

Looking back at the statistical decisions that we learned earlier, that is, Bayesian decision making, it can be easily divided into two steps, first, based on a sample of the PDF estimates, and then based on the estimated PDF to find the classification surface, it is often called two-step Bayesian decision. If we can get a good estimate of the PDF model, we can always use Bayesian to achieve two or even more classes of optimal classification, but in many cases, it is not easy to accurately estimate the PDF model, especially when the sample has high dimensional feature space, and the number of samples is not enough, in essence, The real purpose of pattern recognition is not to estimate the PDF model, but to find a variety of dividing lines or interfaces in the feature space. Therefore, if the classification surface can be obtained directly from the sample, is it possible to omit the estimation of the PDF model and return to the essence, the answer is yes, this blog will learn to learn about the direct design of the classifier based on the knowledge.

2. Three basic elements of the design classifier

Directly designing a classifier based on a sample requires the following three basic elements to be identified:

A. The type of discriminant function, that is, from what discriminant function (set) to find the classification surface;

B. The target or criterion of the classifier design, after determining the criteria, the classifier design is to select the optimal function under the criterion from the definite discriminant function (set) according to the sample, in general, to determine some specific parameters in the function;

C. The first two elements are OK, and the rest is to design algorithms that can search for the best parameters;

In general, is the discriminant function, discriminant criteria and optimization algorithm, in order to express concise, the above three elements can be described in mathematical form: in the discriminant function (set) to determine the undetermined parameters, so that the criteria function to maximize or minimize.

3. What is a linear classifier?

Different discriminant functions, different criteria, and different optimization algorithms all determine the design method of different classifiers, in which the discriminant function is the most important, because the discriminant function is that we want to find the best classification surface according to the categories of the samples, and the classification is basically a good solution , when the discriminant function is a linear function, so the design of the classifier is called a linear classifier or linear discriminant method, the linear classifier is the simplest classifier, in general, the linear classifier can only be suboptimal classifier, but because of its simple design, Moreover, in some cases (for example, the sample distribution obeys normal distribution and all kinds of covariance matrices are equal), the discriminant function can be the best classifier in the sense of minimum error rate or minimum risk, so it is more widely used, especially in the case of finite samples, even better than the nonlinear classifier.

4. What is a linear discriminant function?

First, the general expression of discriminant function is given, and two kinds of cases are:

(1)

There are a number of categories where: (2),C stands for C.

For the sake of simplicity, two types of cases are still used for derivation. Equation (1), X is the D-dimensional sample eigenvector, also known as the sample vector, W is the weight vector, respectively, is represented as follows:

And W0 is a constant value, called the threshold right. For two types of problems, the following decision rule can be used: order, if G (x) >0, then the Class 1, if g (x) <0, then the Class 2, if g (x) = 0, then the arbitrary class or reject. So the g (x) =0 equation defines a decision surface (or classification surface) that separates the two types of points, and when g (x) is linear, the decision surface is a super-plane (Hyper plane). The following is a geometric derivation of the discriminant function:

Assuming there are two samples X1 and x2, which fall on the decision surface H, then there are:

Obviously, W is the normal vector of the classification surface H, it determines the direction of the decision surface H, so for an h that is divided into two half-plane R1 and R2, when X falls in R1, the normal vector w is pointing to the R1, that is, all the sample x in R1 is on the positive side of the categorical surface H, so all the sample x in R2 is on the negative side of H,

At this point, the linear discriminant function g (x) can be regarded as an algebraic measure of the distance from a certain point x to the classification plane h in the sample feature space, and the distance vector of the sample X feature point to H in the R1 is R, which is obtained according to the vector properties:

Therefore, the above formula into our general formula (1), Get:

Therefore, when the sample x=0 (origin), the distance from the origin to the classification plane can be calculated:

Or

It is easy to know that the value of W0 determines the position of the classification surface H, if w0>0, then the origin is in the positive side of the classification surface,w0<0, the origin is on the negative side of H;W0=0, which indicates that the discriminant function is homogeneous and h is over the origin point.

As can be seen from the above deduction, the discriminant function g (x) is actually a sample point x to the classification of the algebraic distance h, when x on the H positive side,g (x) >0; when x on the H negative side,g (x) <0; when x on H, G (x) = 0.

Patterns Recognition (Pattern recognition) Learning Notes (vii)--linear classifier and linear discriminant function