Decision Tree (ID3 for nominal data)

Advantages:
The computational complexity is not high, the output is easy to understand, the median value is not sensitive, and the irrelevant characteristic data can be processed.
Disadvantages:
May cause an over-matching problem.
Applicable data type:
Numerical and nominal type
Information Gain
The biggest principle of partitioning datasets: getting out of order with unordered data
The change in information after the data set is divided is known as information gain
The feature that gets the most information gain is the best choice of partitioning the data set

calculate Shannon entropy Code

def calcshannonent (DataSet):
    labelcounts = {}
    numentries = Len (DataSet) for
    Featvec in DataSet:
        CurrentLabel = featvec[-1]
        if CurrentLabel not in Labelcounts.keys ():
            labelcounts[currentlabel]=0
        Labelcounts[currentlabel]+=1
    shannoent = 0.0 for
    key in labelcounts:
        prob = float (Labelcounts[key])/ NumEntries
        Shannoent-= Prob*log (prob,2)
    return shannoent

The higher the entropy, the more data is mixed.
Partitioning data sets

def splitdataset (dataset,axis,value):
    retdataset = [] for
    Featvec in DataSet:
        if featvec[axis] = = value:
        #将划分依据从集合中删掉
            Reducedfeatvec = Featvec[:axis]
            reducedfeatvec.extend (featvec[axis+1:])
            Retdataset.append (Reducedfeatvec)
    return Retdataset

Choose the best way to partition your data sets

def choosebestfeaturetosplit (DataSet):
    numfeatures = Len (dataset[0])-1
    baseentropy = calcshannonent (DataSet )
    Bestinfogain = 0.0;bestfeature =-1 for
    I in range (numfeatures):
        featlist = [Example[i] For example in DataS ET)
        uniquevals = set (featlist)
        newentropy = 0.0 for
        value in uniquevals:
            subdataset = Splitdataset ( Dataset,i,value)
            prob = Len (subdataset)/float (len (dataSet))
            newentropy + = prob*calcshannonent (subdataset)
        Infogain = baseentropy-newentropy
        if (Infogain > Bestinfogain):
            bestinfogain = Infogain
            Bestfeature = i
    return bestfeature

Recursive build decision tree

 #当用完了所有的特征是还不能明确分类, then majority vote Def majorcnt (classlist): ClassCount = {} for vote in Classlist : If vote not in Classcount.keys (): Classcount[vote] = 0 Classcount[vote] + = 1 Sortedclasscount = Sorte D (Classcount.iteritems (), key = Operator.itemgetter (1), reverse = True) return sortedclasscount[0][0] 

def createtree (dataset,labels):
    classlist = [Example[-1] For example in DataSet]
    if Classlist.count (classlist[ 0]) = = Len (classlist):
        return classlist[0]
    if Len (dataset[0]) ==1:
        return majorcnt (classlist)
    Bestfeat = Choosebestfeaturetosplit (dataSet)
    Bestfeatlabel = labels[bestfeat]
    mytree = {bestfeatlabel:{}}
    Del (Labels[bestfeat])
    featvalues = [Example[bestfeat] For example in DataSet]
    uniquevals = set (Featvalues)
    for value in Uniquevals:
        #由于函数传参数使用的引用, instead of changing the contents of the original list, replace sublabels with the new list
        = labels[:]
        mytree[ Bestfeatlabel][value] = Createtree (
                splitdataset (dataset,bestfeat,value), sublabels)
    return mytree

Storage Decision Tree
Since it takes time to compute the decision tree, we can store the resulting decision tree and take it out when it is used, without having to repeat the calculation.