Analysis and implementation of AdaBoost algorithm

Advantages and disadvantages of AdaBoost (adaptive boosting,adaptive Boosting) algorithm algorithm:

• advantages: Low generalization error rate, easy coding, can be used in most of the classifier, no parameter adjustment
• cons: Sensitive to outliers.

Meta-algorithm (meta algorithm)

In the classification problem, we may not just want to use a classifier, we will consider the combination of classifiers, this method is called the Integration Method (Ensemble) or meta-algorithm . There are many forms of meta-algorithms, which can be integrated with different algorithms or different sets of algorithms.

Two integrated approaches (bagging & boosting)

• The bagging method is also called the Bootstrap aggregation method (bootstrap aggregating). The idea is that a new dataset is randomly sampled from the dataset, then trained with a new data set, and the final result is that the new dataset forms the largest category in the classifier. such as 1000 samples from a data set with a back-up sampling 5,000 times, get 5 new training sets, the algorithm is used in the five training sets to obtain five classifiers.
• Boosting is a method of obtaining results through serial training, the weight of each classifier in bagging is the same, and the weight of classifier in boosting is related to the success of the previous round.

AdaBoost

is one of the most used boosting, the idea is that the next iteration, the weight of the last successful sample is reduced, the weight of the failure increases. Weight Change Mode:

Alpha (classifier weight) Change:
Data weight change:correctly categorized words:Error classification words to realize the idea:

The AdaBoost algorithm is implemented to promote the weak classifier into a strong classifier, so here we first have a weak classifier, the code uses a single-layer decision tree, which is the most used weak classifier, and then we can construct a strong classifier based on the weak classifier

Function:

stumpClassify(dataMatrix,dimen,threshVal,threshIneq)
single-layer decision tree classifier, according to the input value and threshold to compare the output results, because it is a single-layer decision tree, so can only compare the data one dimen value
buildStump(dataArr,classLabels,D)
constructs a single-layer decision tree, this part of the construction of the idea and the previous decision tree is the same, but the evaluation system here is not entropy but weighted error rate, where the weighting is through the data of the weight D to achieve, each build weight will be different from the previous classification results. Information about the returned single-layer decision tree exists in the dictionary structure for easy next use
adaBoostTrainDS(dataArr,classLabels,numIt=40)
AdaBoost's training function is used to combine a bunch of single-layer decision trees to form results. By constantly adjusting the alpha and D to make the error rate nearly 0, or even eventually reach 0
adaClassify(datToClass,classifierArr)
The classification function, Dattoclass, is the data to be classified, and the final result is weighted according to the classification results of a heap of single-layer decision trees generated.

1. #Coding=utf-8 fromNumPyImport*defloadsimpledata (): Datamat= Matrix ([[1., 2.1],        [2., 1.1],        [1.3, 1.],        [1., 1.],        [2., 1.]]) Classlabels= [1.0,1.0,-1.0,-1.0,1.0]    returnDatamat, Classlabelsdefstumpclassify (DATAMATRIX,DIMEN,THRESHVAL,THRESHINEQ): Retarry= Ones (Shape (datamatrix) [0],1))    ifThreshineq = ='LT': Retarry[datamatrix[:,dimen]<= Threshval] = 1.0Else: Retarry[datamatrix[:,dimen]> Threshval] = 1.0returnRetarry#d is the weight vectordefBuildstump (dataarr,classlabels,d): Datamatrix=Mat (Dataarr) Labelmat=Mat (classlabels). T M,n=shape (datamatrix) numsteps= 10.0#traverse on all possible values of a featureBeststump = {}#information for storing a single-layer decision treeBestclasest = Mat (Zeros (m,1)) Minerror=inf forIinchRange (N):#Traverse all featuresRangeMin =datamatrix[:,i].min () RangeMax=Datamatrix[:,i].max () stepsize= (rangemax-rangemin)/numsteps forJinchRange ( -1,int (numsteps) +1):             forInequalinch['LT','GT']: Threshval= (RangeMin + float (j) * stepsize)#get the valve value                #classification according to valve valuePredictedvals =stumpclassify (datamatrix,i,threshval,inequal) Errarr= Mat (Ones (m,1)) Errarr[predictedvals= = Labelmat] =0 Weightederror= d.t * Errarr#The weights of the different samples are not the same.                #print "Split:dim%d, Thresh%.2f, Thresh ineqal:%s, the weighted error is%.3f"% (I, Threshval, inequal, Weighteder ROR)                ifWeightederror <Minerror:minerror=Weightederror bestclasest=predictedvals.copy () beststump['Dim'] =I beststump['Thresh'] =Threshval beststump['Ineq'] =inequalreturnbeststump,minerror,bestclasestdefAdaboosttrainds (dataarr,classlabels,numit=40): Weakclassarr=[] M=shape (Dataarr) [0] D= Mat (Ones ((m,1))/m)#initialize all samples with the same weightsAggclassest = Mat (Zeros ((m,1)))#estimated values for each data point     forIinchRange (numit): Beststump,error,classest=buildstump (dataarr,classlabels,d)#calculation Alpha,max (error,1e-16) ensures that there is no error when no exception occurs except for 0 overflow        #Alpha Represents the weight of this classifier, the lower the error rate, the higher the classifier weightAlpha = float (0.5*log (1.0-error)/max (error,1e-16)) beststump['Alpha'] =Alpha Weakclassarr.append (beststump) expon= Multiply ( -1*alpha*mat (classlabels). T,classest)#exponent for D Calc, getting messyD = Multiply (D,exp (expon))#Calc New D for next iterationD = d/d.sum ()#Calc Training error of all classifiers, if this is 0 quit for loop early ( use break)Aggclassest + = alpha*classest#print "Aggclassest:", aggclassest.tAggerrors = Multiply (sign (aggclassest)! = Mat (classlabels). T,ones ((m,1)) Errorrate= Aggerrors.sum ()/mPrint "Total Error:", ErrorrateifErrorrate = = 0.0:              Break    returnWeakclassarr#Datatoclass represents the point or set of points to classifydefadaclassify (Dattoclass,classifierarr): Datamatrix= Mat (Dattoclass)#Do stuff similar to last aggclassest in Adaboosttraindsm =shape (Datamatrix) [0] Aggclassest= Mat (Zeros (m,1)))     forIinchRange (len (Classifierarr)): Classest= Stumpclassify (datamatrix,classifierarr[i]['Dim'], classifierarr[i]['Thresh'], classifierarr[i]['Ineq'])#Call Stump classifyAggclassest + = classifierarr[i]['Alpha']*classestPrintaggclassestreturnSign (aggclassest)defMain (): Datamat,classlabels=loadsimpledata () D= Mat (Ones (5,1))/5) Classifierarr= Adaboosttrainds (datamat,classlabels,30) T=adaclassify ([0,0],classifierarr)PrintTif __name__=='__main__': Main ()

From for notes (Wiz)

Analysis and implementation of AdaBoost algorithm