Bootstrap aggregating (Bagging) & Boosting

Booststrap aggregating (some local translation: Guided aggregation), which is commonly known as bagging. It is defined as a meta-algorithm for improving the stability and accuracy of machine learning algorithms, which is often used in statistical classification and regression.

Boosting is defined in Wiki as a machine learning meta-algorithm that is mainly used to reduce deviations (Bias) and also to reduce variance (Variance), and is a family of machine learning algorithms that transform weak learners into strong learners. A problem originally raised by Kearns and Valiant (1988,1989) was developed: Can a set of weak learners create a single Stron G Learner? 1996, Breiman

Both are meta-algorithms and are applied to combinatorial learning (Ensemble learning).

Bagging's strategy:

1. Given a size of D training set D;

2. Bagging the M new training set DI, the size of each di is n ', by means of a repeatable uniform sampling from D (by sampling to D uniformly and with replacement);

3. The resulting m new training set, the distribution of regression or classification, to obtain m final results;

4. The final result is obtained from M results, for regression: The average of M results; for classification: M results are voted, the number of votes is the final classification value.

Boosting's strategy:

1. On the training set, train a weak classifier;

2. Add a weak classifier for classification learning:

A. In this process, the weights are adjusted from the previous results, and the training concentration data is re-weighted: adding weights to the wrong sample data, and reducing the weights for the correctly categorized sample data (some boosting algorithms will lower the weight of the repeatedly divided samples, such as BrownBoost and boost by majority);

B. On the training set after weight adjustment, the learning training of weak classifier is carried out;

3. Iteration Step 2

The difference between the two:

(1) from the resampling angle: bagging samples are resampled; In contrast, boosting resampling is not a sample, but a sample distribution, a right to a properly categorized sample, a weighted sample of the wrong classification (which is usually the sample near the classification boundary), The last classifier is a weighted combination of multiple weak classifiers (linear overlay), AdaBoost is the method of representation.

(2) from the final target angle: Bagging can be regarded as the effect of improving the unstable classifier, while the boosting ultimate goal is to improve the classification accuracy of the weak classifier.

Bootstrap aggregating (Bagging) & Boosting