Theoretical study note _ Decision Tree Learning

Decision Tree Learning-machine learning Chapter 2

I. Representation

Ii. applicable issues

1. The instance is a property-value pair.

2. the target function is a discrete output (generally binary, which can be extended to multiple or even continuous values)

3. description to be extracted (relative to the candidate clearing method) eg :( weather = sunny, humidity = normal) V (weather = overcast)

4. robustness and missing attribute values

III. Basic Learning Algorithms

1. Basic concepts:

Entropy:

Entropy indicates the purity of the sample. If the sample S is all positive, the entropy is 0; half is positive, half is negative, and the entropy is 1. This is consistent with the interpretation in Information Theory (the binary digits required by any member in the Encoding S). The larger the uncertainty of the sample, the larger the entropy.

The more general sample s has C categories:

Information Gain: A certain attribute a (with K values) is used to divide sample s into K classes, and calculate the entropy of S for these K classes respectively, calculate the decrease of Entropy of S directly based on the sum of weights:

2. Basic algorithm (ID3 algorithm) Process

3. Suppose space search

The ID3 algorithm also fits the assumption of the training sample from the hypothetical space (all possible decision trees) and leads the search through information gain.

Advantages and disadvantages of ID3 algorithm:

A) assume that the space is complete.

B) compared with the candidate elimination algorithm of the deformation space, ID3 can only store a single current hypothesis, that is, ID3 algorithm does not know other decision trees that are consistent with the current sample.

C) the ID3 algorithm does not trace back to this attribute after selecting attribute tests, so it is easy to produce a local optimal instead of a global optimal solution.

D) ID3 uses the sample statistical information to refine the hypothesis to reduce the impact of individual error samples.

4. Inductive bias of Decision Tree Learning

Give priority to decision trees with shorter values and give priority to decision trees with attributes with higher information gain near the root node.

The bias of the ID3 algorithm comes from the search policy (preferably offset). the offset of the candidate elimination algorithm comes from the definition of the search space (limited offset ).

Occam Razor: give priority to the simplest assumption of Fitting Data

5. Decision Tree Learning FAQs

A) overfitting:

Cause: Incorrect samples and a few coincidence patterns

Solution: Stop tree growth ahead of schedule

Post pruning method

Decision tree Scale Method: use different test sets and clear measurement criteria

Specific Method:

Trim with lower error rate (delete a node, if the trimmed tree does not change the performance of the test set)

Trim rules (create a decision tree and convert it into a rule set. For each rule, delete the rule prefix that can improve precision)

C4.5 conservative estimation (not understandable)

B)
Merge continuous attribute values

Select the threshold C, and convert the attribute to a Boolean value based on whether the attribute value is greater than C.

Method: First, attribute values are sorted in order, and then the intermediate values of adjacent instance attributes with different target classifications are used as a candidate threshold. for all candidate thresholds, the maximum candidate threshold value is calculated.

C) Other metrics selected for attributes-information gain ratio

Problem: if one of the attributes is a time attribute (and each sample value is different), a decision tree that separates all samples can be constructed only based on the time attribute.

Solution: Add a penalty factor

For example, if the time attribute has n values and all samples are split, the split information value is log2n. If attribute a has two values, the split information value is 1, obviously, attribute A has a higher score.

D) Missing attribute values

Processing: Fill n samples with the maximum value of this attribute, or fill the value with the maximum number of occurrences of this attribute in this sample category. Or fill in different values based on the value ratio of this attribute.

E) properties with different processing costs

Some attributes are expensive to obtain (or tissue slice check> body temperature). Therefore, we recommend that you use decision trees with low costs as much as possible. The method is as follows: