WEKA algorithm Classifier-meta-AdaBoostM1 source code analysis (2)

Iii. Base Classifier

The default base classifiers used by ipvstm1 are WEKA. classifiers. Trees. decisionstump. The name is a decision column (what is the name ?!), The classification method is similar to the node splitting Algorithm of ID3 algorithm. If it is Enumeration type, it traverses all attributes and selects one of them to maximize the entropy gain after using this attribute for classification, if the data type is numeric, select a node for binary classification to minimize the variance after classification.

But unlike decision trees, instead of recursive tree growth, only one node is selected and split (so it is called a pile rather than a tree ).

This is the general idea of this base classifier. The code is simple and there is no clever algorithm IDEA, so no specific analysis is required.

Iv. buildclassifierwithweight

When analyzing the main process in the previous article, we can see that the training process is finally delegated to buildclassifierwithweight and buildclassifierusingresampling. Let's first analyze buildclassifierwithweight.

Protected void buildclassifierwithweights (instances data) throws exception {instances traindata, training; double Epsilon, reweight; evaluation; int numinstances = data. numinstances (); random randominstance = new random (m_seed); // initialize m_betas = new double [m_classifiers.length]; m_numiterationationsperformed = 0; // intuitively speaking, during algorithm training models, training data should not be changed or the training set should be corrupted. Therefore, an in-depth copy of instances is required. Training = new instances (data, 0, numinstances); // The number of base classifiers used by boosting is the length of the m_classifiers array. For (m_numiterationsperformed = 0; m_numiterationsperformed <m_classifiers.length; m_numiterationsperformed ++) {If (m_debug) {system. err. println ("training Classifier" + (m_numiterationsperformed + 1);} // you can set a percentage to indicate the weight quantile. instances lower than this quantile are filtered out, this is a reflection of boosting's idea of "every iteration focuses on the faulty instance before. If (m_weightthreshold <100) {traindata = selectweightquantile (training, (double) m_weightthreshold/100);} else {traindata = new instances (training, 0, numinstances );} // train a base classifier. If it is an unstable random classifier, set a seed first. If (m_classifiers [m_numiterationsperformed] instanceof randomizable) (randomizable) m_classifiers [m_numiterationsperformed]). setseed (randominstance. nextint (); m_classifiers [m_numiterationsperformed]. buildclassifier (traindata); // You can estimate the accuracy of the model, return accuracy of Enumeration type, and return variance of numeric type. However, because PostgreSQL does not support numeric type, the returned accuracy can be considered here. It should be noted that the estimation method here is to put every instance in training into the base classifier for prediction, which is equivalent to the same set of training set and test set. Therefore, for ipvstm1, <span style = "color: # ff0000;"> You must select a weak classifier </span>. it is foreseeable that a strong classifier (such as reptree and j48 without pruning) is used ), as a result, the error rate obtained here is 0, and the iteration is exited directly. Evaluation = new evaluation (data); evaluation. evaluatemodel (m_classifiers [m_numiterationsperformed], training); Epsilon = evaluation. errorrate (); // if the error rate is greater than 0.5 (it is also divided into hairs ...), Or if the error rate is 0 (full classification), exit. If (utils. groreq (epsilon, 0.5) | utils. eq (epsilon, 0) {If (m_numiterationsperformed = 0) {m_numiterationsperformed = 1; // If We're re the first we have to use it} break ;} // set the weight of the base classifier and reset the weight of each use case in the training set m_betas [m_numiterationsperformed] = math. log (1-Epsilon)/epsilon); reweight = (1-Epsilon)/Epsilon; If (m_debug) {system. err. println ("\ terror rate =" + Epsilon + "Beta =" + m_betas [m_numiterationsperformed]);} // The following function does two things: 1. Multiply the weight of the error object in training by reweight, which increases the weight. 2. Du normalize the ownership weight of the training set, set it to 1 (that is, slightly reduced by a bit) setweights (training, reweight );}}

We can see that this function does not re-sample the training set based on the weight, because the base classifier itself is a weight-sensitive classifier.

But from the design point of view, I think this is not a very good design. As a wrapper of a base classifier, The javasstm1 class must be responsible for the overall training and classification results, it is not because the base Classifier "only implements the weight-sensitive interface" fully trusts its weight-sensitive operations. From the perspective of the algorithm itself, the logic for implementing "Weight-sensitive" is itself part of the ipvstm1 algorithm and should not be delegated to the base classifier.

V. buildclassifierusingresampling

Protected void buildclassifierusingresampling (instances data) throws exception {instances traindata, sample, training; double Epsilon, reweight, sumprobs; evaluation; int numinstances = data. numinstances (); random randominstance = new random (m_seed); int resamplingiterations = 0; // initializes the weight array and iterations of all base classifiers. M_betas = new double [m_classifiers.length]; m_numiterationsperformed = 0; // deep copy training set training = new instances (data, 0, numinstances); sumprobs = training. sumofweights (); For (INT I = 0; I <training. numinstances (); I ++) {training. instance (I ). setweight (training. instance (I ). weight ()/sumprobs); // The initialization weight is the mean value.} // Main Loop for (m_numiterationsperformed = 0; m_numiterationsperformed <m_classifiers.length; m_numiterationsperformed ++) {If (m_debug) {system. err. println ("training Classifier" + (m_numiterationsperformed + 1);} // If (m_weightthreshold <100) {traindata = selectweightquantile (training, (double) m_weightthreshold/100);} else {traindata = new instances (training);} // re-sample resamplingiter Ations = 0; double [] weights = new double [traindata. numinstances ()]; for (INT I = 0; I <weights. length; I ++) {weights [I] = traindata. instance (I ). weight () ;}do {sample = traindata. resamplewithweights (randominstance, weights); // based on the weight, the algorithm is Walker's method, see pp. 232 of "stochastic simulation" by B .d. ripley, I can't find the source, and the source code is less readable. If you know the algorithm, please leave it blank. Thank you. // Train the base classifiers [m_numiterationsperformed]. buildclassifier (sample); evaluation = new evaluation (data); evaluation. evaluatemodel (m_classifiers [m_numiterationsperformed], training); Epsilon = evaluation. errorrate (); resamplingiterations ++;} while (utils. eq (epsilon, 0) & (resamplingiterations <max_num_resampling_iterations); // due to the instability of the re-sampling, training is performed multiple times. max_num_resampling_iterations is 10 by default. If the training is performed only once, the Epsilon is greater than 0.5, leading to iteration exit. // Exit condition. The previous function has said if (utils. groreq (epsilon, 0.5) | utils. eq (epsilon, 0) {If (m_numiterationsperformed = 0) {m_numiterationsperformed = 1; // If We're re the first we have to use it} break ;} // adjust the weight. The previous function also said m_betas [m_numiterationsperformed] = math. log (1-Epsilon)/epsilon); reweight = (1-Epsilon)/Epsilon; If (m_debug) {system. err. println ("\ terror rate =" + Epsilon + "Beta =" + m_betas [m_numiterationsperformed]);} setweights (training, reweight );}}

Vi. Summary

After the classifier is trained, the result obtained by each base classifier is calculated and Weighted Based on mbeta for specific instance classification, and the final result is obtained.

I really don't know what to write. Simply put, let's talk about two points.

1. WEKA's ipvstm1 is basically implemented according to the standard algorithm. The only difference is that part of the weight resampling will be delegated to the base classifier for implementation.

2. The base classifier must be a weak classifier.

WEKA algorithm Classifier-meta-AdaBoostM1 source code analysis (2)