Boosting, AdaBoost

1, boosting

The boosting method is a method to improve the accuracy of weak classification algorithms by constructing a series of predictive functions and then combining them into a predictive function in a certain way. He is a framework algorithm, mainly through the operation of the sample set to obtain a subset of samples, and then the weak classification algorithm on the sample subset training to generate a series of base classifiers.

Before the boosting algorithm was produced, there were two more important methods to integrate multiple classifiers into one classifier, namely the Boostrapping method and the bagging method.

The main process of 1.1 bootstrapping method

i) sample n samples repeatedly from a sample set D

II) Statistical learning for the set of sub-samples per sample to obtain the hypothesis Hi

III) combine several assumptions to form the final hypothetical Hfinal

IV) Use the final assumptions for specific classification tasks

The main process of 1.2 bagging method

i) training classifier from the whole sample set, sampling n < n samples for sampling set training classifier Ci

II) the classifier to vote, the final result is the winner of the classifier vote

However, both of these methods simply combine the classifiers, and in fact, do not play the power of the classifier combination. Until 1989, Yoav Freund and Robert Schapire proposed a feasible method of combining weak classifiers into strong classifiers.

Schapire also proposes an early boosting algorithm, the main process is as follows:

i) from the sample overall set D , do not put back the random sampling n1 < n samples, get set D1 Training weak classifier C1

II) Extract n2 < n samples from the overall sample set D , which are combined into half of the samples that were incorrectly classified by C1. Get sample Set D2 training weak classifier C2

iii) Samples ofC1 and C2 in the collection of D samples, composition D3 Training weak classifier C3

IV) vote with three classifiers to get the final classification result

By the year 1995, Freund and Schapire proposed the present AdaBoost algorithm.

2, AdaBoost

The main framework can be described as:

i) iterate multiple times, update sample distribution, find the optimal weak classifier under current distribution, calculate the error rate of weak classifier

II) A weak classifier that aggregates multiple workouts

Now, the boost algorithm has a great development, there are a lot of other boost algorithms, such as: Logitboost algorithm, gentleboost algorithm and so on.

3. Proof of convergence of adaboost

The core of the entire proof is:

, which represents the total number of samples, representing the total number of weak classifiers, and the error rate for each level of the weak classifier.

Proof Process:

If so, then. So get the formula.

At this point, see the adaboost error rate limit, the next goal is to make this limit as small as possible!

In the original AdaBoost algorithm, the H domain is { -1,1}, and the question is how to find the best

For the original AdaBoost, the previous article discussed its h is "dead", lost the "bargaining" room, and in real adaboost not "dead".

Deduction process ppt download.

Reference: http://blog.163.com/f_rock/blog/static/1947961312011102810164354/

4, Gentle AdaBoost

Reference: http://blog.csdn.net/wsj998689aa/article/details/42652827

Boosting, AdaBoost