Python implementation of decision tree

The advantages and disadvantages of decision tree algorithms:

• Advantages: The computational complexity is not high, the output is easy to understand, the middle value is not sensitive to the missing, can deal with irrelevant characteristic data
• Cons: Problems that may cause over-matching
• Applicable data type: Numerical and nominal type

Algorithm idea: 1. The whole idea of decision tree construction:

The decision tree is like a if-else structure, and the result is that you want to generate a tree that can judge the leaf node from the root, but the if-else here will not be what we think we are going to set up, we have to do is provide a way The computer can get the decision tree we need based on this method. The point of this approach is to choose from so many features and choose from the root to the leaf in the best order. If we do this, we can construct a decision tree recursively.

2. Information gain

The biggest principle of partitioning a dataset is to make the unordered data more orderly. Since this involves the orderly and disordered information, it is natural to think about the entropy of information. Here we calculate the entropy of information (another method is the Gini purity). The formula is as follows:

Data needs to meet the requirements:

1 data must be a list of list elements, and all Liebei elements have the same data length
2 The last column of the data or the last element of each instance should be the category label of the current instance

Function:

calcShannonEnt(dataSet)
Calculate the Shannon entropy of the data set, in two steps, the first step to calculate the frequency, the second part calculates Shannon entropy according to the formula
splitDataSet(dataSet, aixs, value)
Divides the data set, divides the values that satisfy the x[aixs]==value, and returns a well-partitioned set (excluding the Aixs attribute used for partitioning, because it is not required)
chooseBestFeature(dataSet)
Choose the best attributes to divide, the idea is very simple is to each attribute is divided, to see which is good. Here a set is used to pick the only element in the list, which is a quick way to
majorityCnt(classList)
Because our recursive build decision tree is calculated based on the consumption of the property, there may be a last attribute run out, but the classification is not finished, this time the majority voting method will be used to calculate the node classification
createTree(dataSet, labels)
Build decision trees based on recursion. The label here is more for the name of the classification feature, for better looking and later understanding.

1. 1 #Coding=utf-82 Importoperator3  fromMathImportLog4 Import Time5 6 defCreateDataSet ():7dataset=[[1,1,'Yes'],8[1, 1,'Yes'],9[1, 0,'No'],Ten[0, 1,'No'], One[0, 1,'No']] ALabels = ['No surfaceing','Flippers'] -     returnDataSet, Labels -  the #Calculate Shannon Entropy - defcalcshannonent (dataSet): -NumEntries =Len (dataSet) -Labelcounts = {} +      forFeavecinchDataSet: -CurrentLabel = feavec[-1] +         ifCurrentLabel not inchlabelcounts: ALabelcounts[currentlabel] =0 atLabelcounts[currentlabel] + = 1 -Shannonent = 0.0 -      forKeyinchlabelcounts: -Prob = float (Labelcounts[key])/numentries -Shannonent-= prob * log (prob, 2) -     returnshannonent in  - defSplitdataset (dataSet, axis, value): toRetdataset = [] +      forFeatvecinchDataSet: -         ifFeatvec[axis] = =Value: theReducedfeatvec =Featvec[:axis] *Reducedfeatvec.extend (featvec[axis+1:]) $ retdataset.append (Reducedfeatvec)Panax Notoginseng     returnRetdataset -      the defChoosebestfeaturetosplit (dataSet): +Numfeatures = Len (dataset[0])-1#because the last item in the dataset is a label ABaseentropy =calcshannonent (DataSet) theBestinfogain = 0.0 +Bestfeature =-1 -      forIinchRange (numfeatures): $Featlist = [Example[i] forExampleinchDataSet] $Uniquevals =Set (featlist) -Newentropy = 0.0 -          forValueinchuniquevals: theSubdataset =Splitdataset (DataSet, I, value) -Prob = Len (subdataset)/float (len (dataSet))WuyiNewentropy + = prob *calcshannonent (subdataset) theInfogain = baseentropy-newentropy -         ifInfogain >Bestinfogain: WuBestinfogain =Infogain -Bestfeature =I About     returnbestfeature $              - #because our recursive build decision tree is calculated based on the consumption of the attribute, there may be a last attribute run out, but the classification - #still not finished, this time will use a majority of votes to calculate the node classification - defmajoritycnt (classlist): AClassCount = {} +      forVoteinchclasslist: the         ifVote not inchClasscount.keys (): -Classcount[vote] =0 $Classcount[vote] + = 1 the     returnMax (ClassCount) the      the defcreatetree (DataSet, labels): theClasslist = [Example[-1] forExampleinchDataSet] -     ifClasslist.count (Classlist[0]) ==len (classlist):#the same category stops dividing in         returnClasslist[0] the     ifLen (dataset[0]) = = 1:#all features are out of use the         returnmajoritycnt (classlist) AboutBestfeat =choosebestfeaturetosplit (DataSet) theBestfeatlabel =Labels[bestfeat] theMytree ={bestfeatlabel:{}} the     del(Labels[bestfeat]) +Featvalues = [Example[bestfeat] forExampleinchDataSet] -Uniquevals =Set (featvalues) the      forValueinchuniquevals:BayiSublabels = labels[:]#to copy the contents of the original list without changing it. theMytree[bestfeatlabel][value] =Createtree (Splitdataset (DataSet, the bestfeat, value), sublabels) -     returnmytree -      the defMain (): theData,label =CreateDataSet () theT1 =Time.clock () theMytree =Createtree (Data,label) -t2 =Time.clock () the     Printmytree the     Print 'Execute for', t2-T1 the if __name__=='__main__':94Main ()

From for notes (Wiz)

Python implementation of decision tree