Quick understanding of bootstrap,bagging,boosting,gradient boost-Three concepts

 

1 booststraping: It means to rely on your own resources, called self-help method, which is a kind of sampling method that has put back, it is an important statistic method of estimating statistic variance in nonparametric statistics and then making interval estimation.

Its core ideas and basic steps are as follows:

(1) using the resampling technique to extract a certain quantity (own given) sample from the original sample, this process allows repeated sampling.

(2) Calculate the statistic T based on the extracted sample.

(3) Repeat the above n times (generally greater than 1000), get the statistics T.

(4) Calculate the variance of the sample of the n statistic T, and get the statistic variances.

It should be said that Bootstrap is a popular method of modern statistics, small sample effect is good, through the estimation of variance can construct confidence interval.

2 bagging:bootstrap aggregating abbreviation. The Learning algorithm trains multiple rounds, and each round of training set consists of randomly extracting n training samples from the initial training set, a training sample may appear multiple times in a training set

Or does not appear, after training can be obtained a predictive function sequence h_1,h_n, the final prediction function h to the classification of the problem by voting, the regression problem (weighted average good, but not) using a simple average way to discriminate.

Training R Classifier F_i, the other identical between the classifier is the same parameter step. The f_i is obtained by taking n samples from the training set and randomly. For a new sample, use this r classifier to classify, get the most of that category

Is the final category of the sample.

3 Boost: The main one is adaboost (adaptive boosting). The same weights are assigned to each training during initialization, and then the T-wheel is trained on the training set by the 1/n algorithm, and after each training, the training of training failure is assigned

To the larger weight, that is, to let the learning algorithm in the follow-up learning focus on the more difficult training column training (that is, the training of the wrong classification of the sample, again to take out training, see it later dare not), so as to get a forecast

The function sequence h_1,h_m, in which H_i also has certain weights, the predictive function with good prediction effect is significant and vice versa. The final predictive function H uses a weighted voting method for the classification problem, and the weighting average method is applied to the regression problem.

New sample identification.

Similar to the bagging method, but the training is serial, the K classifier is trained to focus on errors in the pre-k-1 classifier, not randomly taking samples, but increasing the weights of the samples that are divided.

4 The difference between bagging and boosting : The main difference between the two is to take the sample in different ways. Bagging adopts uniform sampling, and boosting is sampled according to error rate, so boosting classification accuracy is due to bagging.

The training set selection of baging is random, and the training sets are independent before each other, while the selection of the boosting training sets is related to the learning results of the previous rounds; the bagging of each predictive function has no weight, and boost has the power to weigh;

The functions of bagging can be generated in parallel, while the individual predictive functions of boosting are only sequentially generated.

For extremely time-consuming algorithms like neural networks, bagging can save significant time overhead by parallel. Both baging and boosting can effectively improve the accuracy of classification. In most data sets, boosting is accurate

Sex is higher than bagging. There are some data lumped, boosting will degenerate-overfit. An improved AdaBoost method of boosting thought has good performance in message filtering and text categorization.

5 Gradient Boost (mart,treenet): Boosting is a thought, gradient boosting is a way to implement boosting, and its main idea is that every time a model is built, Is the ladder that created the model loss function before

Direction of decline. The loss function describes the degree of uncertainty of the model, the larger the loss function, the more error-prone the model is. If our model allows the loss function to continue to decline, it means that our model is constantly improving, and the best

The way to do this is to let the loss function fall in the direction of its gradient.

6 Rand Forest: Random forest, using a random way to build a forest, there are many decision trees in the forest, the random forest of each lesson decision tree is not associated. After getting the forest, when there is a new input

Sample entry, let each decision tree in the forest to judge, see this sample belongs to that class, and then see which category is selected more, the prediction of that category.

In the process of building decision trees, it is necessary to pay attention to two points-sampling and complete splitting. The first is the two random sampling process, the random forest to the input data by row, column sampling.

For line sampling, there may be duplicate samples in a playback mode, that is, in the sample collection that is sampled. Assuming that the input sample is N, the sampled sample is also N. This makes it possible for each tree to be trained

The input samples are not all samples, making the over-fitting relatively easy to appear. Then sample, from M feature, select M.

The decision tree is then created in a completely fragmented manner after the sampled data, so that one of the leaf nodes of the decision tree is either unable to continue splitting, or all the samples inside are pointing to the same category. A General decision Tree

There is an important step to pruning, but this does not work here, because the previous two random sampling process guarantees randomness, so even without pruning, it will not over-fitting. Each of the random forests obtained by this algorithm

The trees are very weak, but the combination of them is very powerful. It can be likened to random forests: each decision tree is an expert in a narrow field, so there are many different areas of expertise in the random forest

At home, for a new sample, it can be viewed in different ways, and ultimately by various experts, voting results.

Top

1

Quick understanding of bootstrap,bagging,boosting,gradient boost-Three concepts