Machine learning Algorithm-Bayesian classifier (i)

1. Bayesian decision-making theory

Definition: Bayesian decision-making theory is a basic method to implement decision-making under probabilistic framework. For the classification task, in the ideal case where all probabilities are known, Bayesian decision-making considers how to Select the optimal category tag based on these probabilities and miscalculation losses, and the following is an example of a multi-classification task to explain its rationale.

Conditional risk: Assuming there is a possible category tag in N, that is, Y={c1,c2,..., CN}, Λij is the loss of classifying a sample that is actually labeled CJ as a CI. Based on the posteriori probability P (ci| x) to obtain the desired loss from the classification of the sample x as CI, i.e. "conditional risk" on sample x

This is a condition risk for a single sample, and if the risk of each sample is minimized, the overall risk of all the samples will be minimized, resulting in the Bayesian criteria: to minimize the overall risk, simply select the category tag on each sample that will minimize the conditional risk R (ci|x), i.e.

Note: at this time h* is called Bayesian Optimal classifier, and the corresponding overall risk R (h*) is called Bayesian risk, when the risk is minimal, the performance of the classifier to achieve the best.

Specifically, if the goal is to minimize the classification error rate, the miscalculation loss Λij can be written as:

At this time the conditional risk R (c|x) =1-p (c|x), so the Bayesian optimal classifier minimizing the classification error rate is

that is, for each sample x, the selection can make a posteriori probability p (c| X) the largest category tag.

In order to minimize the risk of decision-making using Bayesian criterion, we first obtain the posteriori probability P (c| x), but it is hard to get directly in real life. From this point of view, machine learning is accomplished by estimating the posteriori probability P (c| ) as accurately as possible based on a limited training sample. x). Broadly speaking, there are two strategies:

(1) Given x, Direct modeling P (c| x) to predict C, thus obtaining a "discriminant model";

(2) the joint probability p (x, C) is modeled first, then P (c| x), the "Generative Model" (Note: Decision Tree, BP Neural network, support vector machine, etc.) can be classified into discriminant model. for a built-in model, consider

Note: P (c) is a "priori" probability, p (x|c) is the class conditional probability of the sample x relative to the class Mark C, or "likelihood"; p (x) is the normalized factor. Given a sample x, P (x) is independent of the class tag,

It is therefore estimated that P (c| x) The problem is translated into how to estimate a priori p (x) and likelihood P (x|c) based on the training data D, and P (c) can be estimated by the frequency of each sample, and the class condition probabilities cannot be directly estimated, The next article explains how to estimate the probability of a class condition.

Machine learning Algorithm-Bayesian classifier (i)