Statistical learning Method –> Decision Tree

Preface:

Purpose: Classification.

Similar to If-then collection

Advantages: Fast speed.

Principle: The loss function is minimized, which is the principle of all machine learning algorithms.

Step:1> Feature Selection 2> decision tree generation 3> decision tree pruning

Decision Tree Model

Internal nodes and leaf nodes, too well understood, without discussion.

If-then Collection

If-then need to ensure that the important point: is mutually exclusive and complete. Very well understood. Complete guarantees that each instance has a choice. Mutual exclusion guarantees that each instance has only one choice.

decision tree and conditional probability distribution

When classifying the nodes, the example of the node is divided into the higher probability of the party.

Decision Tree Learning

The essence of Decision tree Learning is the rules of various classifications. At the very least, ensure that the rules obtained in the classification are inconsistent with the training data.

Decision trees tend to cross-fit (Gbdt also tend to fit). The better the classification feature, the more first it is used. As long as there are no categories of samples, continue to build features to classify.

Decision tree generation corresponds to a local selection. When pruning, we should consider the overall optimal.

The common algorithm has id3,c4.5 CART three kinds.

Feature Selection

Criteria for Feature selection: Information gain and information gain ratio.

Information gain

G (d,a) = H (D) –h (d| A

Choose the largest of G (D,a) as the preferred feature.

Information gain ratio

GR (D,a) = g (d,a)/h (d) H (d) For information gain/training set about the entropy of the characteristic a.

ID3 algorithm

The most typical method of a decision tree. From the root node, select the characteristics of the corresponding maximum information gain as the selected feature. After establishing the sub-nodes, the characteristics of the corresponding maximum information gain are constructed.

There is a situation that may occur. All the features that can build the tree are exhausted. The training data has not yet been completely separated. If this happens, don't try to differentiate. The training data left to see what kind of results, as the result of this node is good.

Another scenario is that you build features, but the information gain is particularly small. Smaller than the tolerable threshold value. It also gives the value of the leaf node in the same way as above.

the generation algorithm of C4.5

The improvement was made on the basis of ID3.

The choice of feature is the information gain ratio.

The rest of the same.

Pruning of decision trees

Pruning is the simplification of the tree.

Pruning principle: Minimizing loss function or cost function. The model selection is made with regularization maximum likelihood estimation.

The loss function is so complex that it is not remembered. It's enough to know the above two sentences.

Cart Algorithm Preface

CART: Classification tree and regression tree.

Walk three steps: Feature selection, spanning tree, pruning the tree.

Statistical learning Method –> Decision Tree