Integrated learning Fazi bagging method and boosting method

I. Integrated learning method (Ensemble Learning)

First, let's take a look at what an integrated learning method is.

① brings together multiple classification methods to improve the accuracy of classification.

(These algorithms can be different algorithms, or they can be the same algorithm.) ）

The ② integrated learning method constructs a set of base classifiers from the training data and then classifies them by voting on the predictions of each base classifier.

③ strictly speaking, integration learning is not a classifier, but a method of combining classifiers.

④ typically the classification performance of an integrated classifier is better than a single classifier

⑤ If you compare a single classifier to a decision maker, the integrated learning approach is equivalent to a decision made by multiple decision makers.

(Integrated learning method illustration)

To master the Integrated learning approach, we will ask the following two questions:

1) How to train each algorithm.

2) How to fuse each algorithm.

Therefore, the bagging method and the boosting method emerge

second, bagging (bagging) method

①bagging is also called self-help aggregation, which is a technique of repeating sampling (with put-back) from data based on a uniform probability distribution.

② a base classifier on each sample generated self-service sample set, polls the trained classifier, and assigns the test sample to the class with the highest ticket.

③ each self-help sample set is as large as the original data

④ have a sample put back, some samples may appear in the same training set multiple times, some may be ignored.

(Bagging method plots)

To give you a better understanding of the bagging approach, here is an example.

X represents a one-dimensional attribute, and Y represents the class label (1 or-1) test condition: When X<=k, y=. ; when X>k, y=. K is the best splitting point.

The following table is the only correct y category that corresponds to the attribute X

5 rounds of random sampling are now conducted with the following results

After each round of random sampling, a classifier is generated

And then merge the five-round classification

Comparing symbols and actual classes, we can find that: In this example, the bagging makes the accuracy rate up to 90%

Thus, summarize the bagging method:

①bagging improves the generalization error by reducing the variance of the base classifier
② its performance depends on the stability of the base classifier, and if the base classifier is unstable, bagging helps to reduce the error caused by random fluctuations in the training data, and if stable, the error of the integrated classifier is mainly caused by the bias of the base classifier.
③ because the probability of each sample being selected is the same, bagging does not focus on training any particular instance of the data set

third, boosting (Ascension) method

①boosting is an iterative process used to adaptively change the distribution of training samples, allowing the base classifier to focus on the hard-to-divide samples

The ②boosting assigns a weight to each training sample and can automatically adjust the weights at the end of each round of ascension process. At first, all the samples were given the same weight 1/n, so that they were chosen as the training possibilities. According to the sample distribution of the training samples, the samples were extracted and the new sample set was obtained. The training set then summarizes a classifier and uses it to classify all the samples in the original data set. At the end of each round of promotion, the weights of the training set samples are updated. Increases the weights of the samples that are incorrectly categorized, reducing the weights of the correctly categorized samples, which allows the classifier to focus on those samples that are difficult to classify in subsequent iterations.

So we can compare bagging and boosting.

①bagging training set is random, each training set is independent, and the selection of boosting training set is not independent, each time the chosen training set depends on the result of the last learning

Each predictive function of ②bagging has no weight, and boosting the weight of the predicted function based on the training error of each training.

The various predictive functions of ③bagging can be generated in parallel, whereas boosting can only be generated sequentially. (Bagging can save a lot of time overhead by parallel training for a very time-consuming learning method such as neural networks)

However, bagging and boosting can be regarded as more traditional integrated learning ideas. Now commonly used random FOREST,GBDT (iterative decision tree), Gbrank is actually more refined, more effective method.