"Machine Learning Algorithm Foundation + Combat series" Decision tree algorithm

A decision tree is a basic classification and regression model, which means that it can be used for both classification and regression. Here, take the classification as an example.
Decision tree Model A tree structure that describes the classification of instances by characteristics, which consists of a root node, several internal nodes, and several leaf nodes. The leaf node corresponds to the decision result, or it can be said to be a class, the internal node represents a property and characteristics.
The learning algorithm of decision tree consists of three steps: Feature selection, decision tree generation, pruning of decision tree

Feature Selection

Feature selection is to choose the characteristics of the training data has a good classification ability, if the selected characteristics of the classification results and the results of random classification is not very different, then it can not be said that this feature has a good classification ability. In terms of experience, throwing away these features will not affect the accuracy of the decision tree learning.
The criteria for General feature selection are the information gain or the information gain ratio we are taking.

Information gain:

First we give the definition of entropy: The uncertainty of the random variable represented by entropy.
Set \ (x\) is a discrete random variable with a finite number of values, with a probability distribution of:\[p (X=x_{i}) =p_{i}, i =,..., n\]
The entropy of the random variable is defined as: \[H (X) =-\sum_{i=1}^{n}p_{i}logp_{i}\]
Normally, the logarithm in the equation is based on 2 or E. If \ (P_{i} = 0, define 0log0=0\). By defining the formula we can tell that entropy depends only on the distribution of X, and not on the value of X. So we can also rewrite this equation:\[H (P) =-\sum_{i=1}^{n}p_{i}logp_{i}\]
The greater the entropy, the greater the uncertainty of the random variable, from which we can know:\ (0\leq H (p) \leq logn\)
Next we give a definition of conditional entropy. With a random variable (x, y), its joint probability distribution is

"Machine Learning Algorithm Foundation + Combat series" Decision tree algorithm