[Reprinted] Easy-to-learn machine learning algorithm-ID3 algorithm of decision tree

I. Overview of decision tree classification algorithms the decision tree algorithm is based on data attributes (or features) and uses attributes as the basis to divide different classes. For example, for the following dataset (Dataset), the first and second columns are attribute (feature), the last column is category tag, 1 indicates yes, and 0 indicates no. The idea of the decision tree algorithm is to classify data based on attributes. For the above data, we can obtain the following decision tree model (decision tree model). First, a part of data is partitioned based on the first attribute, separate the remaining partitions based on the second attribute. Decision Tree algorithms include ID3, C4.5, and cart. Next we will introduce the ID3 algorithm. Ii. Overview of ID3 algorithm ID3 algorithm was first proposed by Quinlan, which is based on information theory and measured by information entropy and information gain, so as to implement data induction and classification. First, the ID3 algorithm needs to solve the problem of selecting features as the criteria for dividing datasets. In the ID3 algorithm, select the attribute with the largest information gain as the current feature for dataset classification. The concept of information gain is described below, and the dataset is continuously divided by selecting features. Secondly, the ID3 algorithm needs to solve the problem of determining the end of the division. There are two cases. The first case is that the divided classes belong to the same class. For example, the leftmost "non-fish" in the class is the data of 5th rows and 6th rows in the dataset; the rightmost "fish" is the 2nd rows and 3rd rows of data in the dataset. Second, there is no attribute available for further splitting. This is the end. Through iteration, we can obtain such a decision tree model. (Basic process of ID3 algorithm) 3. Data Division based on ID3 algorithm is a classification algorithm that measures information entropy and information gain. 1. The concept of entropy mainly refers to the degree of information confusion. The greater the uncertainty of a variable, the greater the entropy value. The entropy formula can be expressed as: Where ,, the probability that a category appears in the sample. 2. Information Gain refers to the change of entropy before and after division. It can be expressed by the following formula: In this formula, it represents the attributes of the sample and a set of values of all attributes. Yes, one of the attribute values is a sample set of values in. Iv. Experiment Simulation 1. data preprocessing the following data is used as an example to implement the ID3 algorithm. (Data-based tables) 2. Experiment results (original data) (partition 1) (partition 2) (partition 3) (final decision tree) Matlab code main program [Plain]View plaincopy

1. % Demo-tree
2. % ID3
4. % Import data
5. % DATA = [, 1;, 1;, 0;, 0;, 0];
7. Data = [0, 2, 0, 0;
8. 0, 2, 0, 0;
9. 1, 2, 0, 0, 1;
10. 2, 1, 0, 0, 1;
11. 2, 0, 1, 0, 1;
12. 2, 0, 1, 1, 0;
13. 1, 0, 1, 1;
14. 0, 1, 0, 0;
15. 0, 0, 1, 0, 1;
16. 2, 1, 1, 0, 1;
17. 0, 1, 1, 1;
18. 1, 1, 1, 1;
19. 1, 2, 1, 0, 1;
20. 2, 1, 0, 0];
22. % Generate decision tree
23. Createtree (data );

Generate Decision Tree [Plain]View plaincopy

1. Function [output_args] = createtree (data)
2. [M, N] = size (data );
3. Disp ('original data :');
4. Disp (data );
5. Classlist = data (:, N );
6. Classone = 1; % record the number of the first class
7. For I = 2: m
8. If classlist (I, :) = classlist (1 ,:)
9. Classone = classone + 1;
10. End
11. End
13. % Categories are all the same
14. If classone = m
15. Disp ('final data :');
16. Disp (data );
17. Return;
18. End
20. % All features used up
21. If n = 1
22. Disp ('final data :');
23. Disp (data );
24. Return;
25. End
27. Bestfeat = choosebestfeature (data );
28. Disp (['bestfeat: ', num2str (bestfeat)]);
29. Featvalues = unique (data (:, bestfeat ));
30. Numoffeatvalue = length (featvalues );
32. For I = 1: numoffeatvalue
33. Createtree (splitdata (data, bestfeat, featvalues (I ,:)));
34. Disp ('-------------------------');
35. End
36. End

Select the feature with the largest information gain [Plain]View plaincopy

1. % Select the feature with the largest information gain
2. Function [bestfeature] = choosebestfeature (data)
3. [M, N] = size (data); % get the dataset size
5. % Number of Statistical Features
6. Numoffeatures = n-1; % The last column is a category.
7. % Original entropy
8. Baseentropy = calentropy (data );
10. Bestinfogain = 0; % initialization information gain
11. Bestfeature = 0; % initialize the best feature bit
13. % Select the best feature bit
14. For j = 1: numoffeatures
15. Featuretemp = unique (data (:, j ));
16. Numf = length (featuretemp); % number of attributes
17. Newentropy = 0; % entropy after Division
18. For I = 1: numf
19. Subset = splitdata (data, J, featuretemp (I ,:));
20. [M_1, n_1] = size (subset );
21. Prob = M_1./m;
22. Newentropy = newentropy + prob * calentropy (subset );
23. End
25. % Gain Calculation
26. Infogain = baseentropy-newentropy;
28. If infogain> bestinfogain
29. Bestinfogain = infogain;
30. Bestfeature = J;
31. End
32. End
33. End

Calculate entropy [Plain]View plaincopy

1. Function [entropy] = calentropy (data)
2. [M, N] = size (data );
4. % Get category items
5. Label = data (:, N );
7. % Processed label
8. Label_deal = unique (Label );
10. Numlabel = length (label_deal );
11. Prob = zeros (numlabel, 2 );
13. % Statistics tag
14. For I = 1: numlabel
15. Prob (I, 1) = label_deal (I ,:);
16. For j = 1: m
17. If label (J, :) = label_deal (I ,:)
18. Prob (I, 2) = prob (I, 2) + 1;
19. End
20. End
21. End
23. % Calculate entropy
24. Prob (:, 2) = prob (:, 2)./m;
25. Entropy = 0;
26. For I = 1: numlabel
27. Entropy = entropy-prob (I, 2) * log2 (prob (I, 2 ));
28. End
29. End

Divide data [Plain]View plaincopy

1. Function [subset] = splitdata (data, axis, value)
2. [M, N] = size (data); % to get the data size to be divided
4. Subset = data;
5. Subset (:, axis) = [];
6. K = 0;
7. For I = 1: m
8. If data (I, axis )~ = Value
9. Subset (I-K, :) = [];
10. K = k + 1;
11. End
12. End
13. End

Http://blog.csdn.net/google19890102/article/details/28611225