Python Sklearn Adaboost

1. AdaBoost Class Library Overview

Scikit-learn in AdaBoost class library is directly, Adaboostclassifier and Adaboostregressor Two, from the name can be seen Adaboostclassifier for classification, Adaboostregressor for regression.

Adaboostclassifier uses the implementation of two AdaBoost classification algorithms, Samme and SAMME.R. The adaboostregressor uses the implementation of the AdaBoost regression algorithm we talked about in our theory, namely ADABOOST.R2.

When we are on the adaboost, we mainly need to adjust the two parts of the content, the first part is to our AdaBoost framework for the assistant, the second part is to our choice of weak classifier for the assistant. The two complement each other. Here are the two classes for AdaBoost: Adaboostclassifier and adaboostregressor from these two parts.

2. Adaboostclassifier and Adaboostregressor frame parameters

Let's take a look at the Adaboostclassifier and adaboostregressor frame parameters first. Most of the framework parameters are the same, we discuss these parameters together, two classes if there are different points we will point out.

1)both Base_estimator:adaboostclassifier and Adaboostregressor have, that is, our weak classifier learner or the weak regression learning device. In theory, you can choose either a classification or a regression learner, but you need to support sample weights. We commonly use the cart decision tree or the neural network MLP. The default is the decision tree, where Adaboostclassifier uses the cart classification tree decisiontreeclassifier By default, and Adaboostregressor uses the cart regression tree decisiontreeregressor by default. Another point to be aware of is that If we choose the Adaboostclassifier algorithm is SAMME.R, then our weak classifier will also need to support the probability of prediction, that is, in the Scikit-learn the corresponding prediction method of weak classification learners in addition to predict also need to have predict_ Proba

2)algorithm: This parameter is only adaboostclassifier. The main reason is that Scikit-learn realizes two kinds of adaboost classification algorithms, Samme and SAMME.R. The main difference between the two is the measurement of the weight of the weak learner, Samme uses the extension of the two-yuan classification adaboost algorithm in our principle, that is, using the classification effect of the sample set as the weak learner weight, and SAMME.R uses the predictive probability size of the sample set classification as the weak learner weight. Since SAMME.R uses a continuous value for probability measures, iterations are generally faster than samme, so the adaboostclassifier default algorithm algorithm value is also SAMME.R. We generally use the default SAMME.R, but it is important to note that the SAMME.R is used, and the weakly categorical learner parameter Base_estimator must limit the use of classifiers that support probabilistic predictions. The samme algorithm does not have this limitation.

3)loss: This parameter only Adaboostregressor has, ADABOOST.R2 algorithm needs to use. There are linear ' linear ', square ' square ' and exponential ' exponential ' three choices, which are linear by default, and are generally sufficient for linear use unless you suspect that this parameter results in a bad fit. The significance of this value is also discussed in the principle, which corresponds to our handling of the error of the first sample of the K-weak classifier, i.e., if it is a linear error, if it is a squared error, then, if it is an exponential error, the maximum error on the training set

4) N_estimators:adaboostclassifier and Adaboostregressor, is the maximum number of iterations of our weak learner, or the largest number of weak learners. Generally speaking, n_estimators is too small, easy to fit, n_estimators too large, and easy to fit, generally choose a moderate value. The default is 50. In the process of actual tuning, we often consider n_estimators and the parameters described below Learning_rate.

5)  learning_rate:   Adaboostclassifier and Adaboostregressor have, that is, the weight reduction factor of each weak learner In the regularization section of the principle we also talked about, plus the regularization term, our strong learner's iterative formula is start the parameter, default is 1.

3. Adaboostclassifier and adaboostregressor Weak learner parameters

Here we discuss the adaboostclassifier and adaboostregressor weak learner parameters, because of the use of different weak learners, the corresponding weak learner parameters are different. Here we only discuss the parameters of the default decision tree weak learner. that is, the cart classification tree decisiontreeclassifier and the cart regression tree decisiontreeregressor.

Decisiontreeclassifier and Decisiontreeregressor parameters are basically similar, in the Scikit-learn Decision Tree algorithm Class Library Use summary This article we have the parameters of these two classes are explained in detail. Here we only take out the parameters of the most important points to pay attention to the parameters of the most significant again to say again:

1)   Maximum number of features considered when dividing max_features:  can use many types of values, The default is "None", which means that all features are considered when dividing, and if "log2" means that most of the partitioning is considered

2) Decision tree maximum deep max_depth: Default can not input, if not input, the decision tree when establishing a subtree does not limit the depth of the subtree. In general, you can ignore this value when there is less data or features. If the model sample size is large and the characteristics are many, it is recommended to limit the maximum depth, depending on the distribution of the data. Commonly used can be valued between 10-100.

3) Minimum number of samples required for internal node re -Partitioning min_samples_split: This value restricts the condition that the subtree continues to divide, and if a node has fewer samples than min_samples_split, it will not continue to try to select the optimal feature for partitioning. The default is 2. If the sample size is small, do not need to tube this value. If the sample quantity is very large, it is recommended to increase this value.

4) Minimum number of leaf nodes min_samples_leaf: This value limits the minimum number of leaf nodes, and if a leaf node is less than the number of samples, it will be pruned along with the sibling nodes. The default is 1, an integer that can enter a minimum number of samples, or a minimum number of samples as a percentage of the total number of samples. If the sample size is small, you do not need to tube this value. If the sample quantity is very large, it is recommended to increase this value.

5) Minimum sample weights and min_weight_fraction_leaf for leaf nodes : This value limits the minimum value of all sample weights and the leaf node, and if it is less than this value, it is pruned along with the sibling nodes. The default is 0, which is to not consider the weight issue. In general, if we have more samples with missing values, or if the classification tree sample has a large variation in the distribution category, we will introduce the sample weights, and we should pay attention to this value.

6) Maximum leaf node number max_leaf_nodes: By limiting the maximum number of leaf nodes, you can prevent overfitting, the default is "None", that is, the maximum number of leaf nodes is not limited. If the limit is added, the algorithm will establish the optimal decision tree in the maximum number of leaf nodes. If the feature is not many, you can not consider this value, but if the features are divided into more, you can limit the specific values can be obtained by cross-validation.

Python Sklearn Adaboost