Statistical learning method Note lifting tree

Ascending tree is an ascending method based on the classification tree or the regression tree as the basic classifier.

Ascension Tree is considered to be one of the best performance methods in statistical learning.

The lifting method actually adopts the addition model ( linear combination of the basis function ) and the forward Step-up algorithm

A lifting method based on decision tree is called an ascending tree

?

Decision tree for classification problem is a two-fork classification tree

Decision tree for regression problem is a two-fork regression tree

?

The lift tree model can be represented as an addition model for the decision tree:

T (x;θm) represents a decision tree;

Θm parameters for the decision tree;

M is the number of trees.

The Ascending tree algorithm adopts forward step algorithm .

first, determine the initial lift tree F0(x) =0, model m on foot is

F m-1 (x) to determine the parameters of the next decision tree for the current model by minimizing empirical risk Θm ,

?

?

The following is a discussion of the lifting tree learning algorithm for different problems, the main difference is the use of the loss function is different.

Including the regression problem of the square error loss function , using the classification problem of exponential loss function

Ascending tree for regression problem:

training data set t = {(x 1 y " Span style= "font-size:9pt" >1 2 ,y 2 ),..., (x n ,y n )}

If the input space x divided into j disjoint area r 1 ,r 2 ,..., R j c J

parameter Θ= {(R1, c1), (R2, C2),..., (r J, CJ)} represents the partition of a tree and the constants on each region .

J is the complexity of the regression tree, that is, the number of leaf nodes .

Use the following forward distribution algorithm:

in the forward step-by-step algorithm, the first m step, given the current model Fm-1(x) , the demand solution

get Theta, that is, the first m the parameters of the tree.

When the square error loss function is used:

The loss becomes:

Over here:

is the residual of the current model fit data ( residual ).

So, for the ascending tree algorithm of regression problem, we simply fit The residuals of the current model.

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

?

Statistical learning method Note lifting tree