GBDT (Gradient Boosted decision Tree)

GBDT, the full name gradient Boosted decision Tree, is a model composed of multiple decision trees, which can be used for classification and regression.

The origin of GBDT the popular way of understanding the advantages and disadvantages of mathematical expression GBDT

The origin of the

GBDT Decision Tree is one of the common models, it uses heuristic search method to find the interval (dividing the space of eigenvector), divided by several kinds, such as information gain, Gini index and so on. One of the common problems in decision Trees is overfitting (which is, of course, a problem for all models, which is more common in decision trees), and for the current training data, the depth of the tree is much, too few, the data is not sufficiently differentiated, and too large it can lead to overfitting. Because these parameters can not be found through the specific mathematical theory, so, in the decision-making tree parameters, often have such doubts: the depth of the tree is how much better, the number of leaves a total number of suitable. There is only one decision tree to make the decision is too thin, there is no way through the decision-making tree to decide. Similarly, with random forests this relies on multiple decision trees and ADA boosting, a strategy that blends multiple classifiers to improve the classification and regression effects.
In this case, the GBDT is generated. Popular Way of understanding first to understand the definition of residuals , GBDT is based on this to build ŷ =y−predict (x), y \hat{y}=y-predict (x) , Y is the label value we want to predict, predict (x) predict (x) is the predictive value of the decision tree, and the residuals are defined as the difference between the model's predicted value and the true value. not as good as many times a small step optimization Now we've tuned to the blue point, and if we're too big for each iteration, we're going to get into another surface all at once, causing the model to fail to converge. Overall, a one-time optimization can lead to a model that cannot converge, but a small optimization can only cause a decrease in convergence speed. GBDT's overall idea is that the combination of the above two, now my decision tree prediction is not allowed, it's okay. I will not deal with it, I leave the rest of the residuals to the next tree to deal with. This is different from AdaBoost, the latter is that I am not allowed to predict this part of the data, I increase the weight of this part of the data, the next model focuses on this part of the data. In general, GBDT left the current decision tree forest unresolved issues (residuals) to the next tree, iterating over it, and ultimately the whole model's predictions are the sum of each decision tree. Mathematical Expression

The

Symbol Description: F (i) f (i) represents the model that is obtained after the iteration of the I, F (i) f (i) represents the first step of the prediction result of the decision tree: initialization. Train the first decision tree, get F (1) F (1), because now there is only one tree, so f (1) =f (1) F (1) =f (1) The second step: residual calculation. According to the present model F (1) f (1) We can calculate the current predicted value (f (x1), F (x2), F (x3),... F (xn)) (f (x_1), F (x_2), F (x_3),... F (X_n)), according to the above description, we calculate residuals can be obtained (Ŷ 1,Ŷ 2,Ŷ 3...Ŷ N) (\hat{y}_1,\hat{y}_2,\hat{y}_3...\hat{y} _n) Step three: Add a new decision tree. Now the second tree's fitting goal is to make some changes, no longer to fit the label to the target, but to fit the residual left before the target (the feature set or unchanged, only the target changes to fit). Fourth step: Update the model. Now that we have a new decision tree, the second thought we mentioned above comes: we are getting closer to the optimal value instead of arriving at once, so we update the model in such a way: F (2) =f (1) +a∗f (2), a (0,1)