Source code analysis of WEKA algorithm Classifier-tree-J48 (I) algorithm and Basic Data Structure

I haven't taken care of my blog for about a year. I don't know where to write the pen again. I want to write it from weka I 've been using recently.

WEKA is a machine learning tool based on Java. It is easy to use and provides graphical interfaces, such as classification, clustering, and frequent item mining, this article mainly describes the j48 algorithm in the classifier Algorithm and its implementation.

I. Algorithms

J48 is based on C4.5 decision tree algorithm, for C4.5 algorithm related information too much, the author here reprinted part (Source: http://blog.csdn.net/zjd950131/article/details/8027081)

C4.5 is a series of algorithms used in classification issues of machine learning and data mining. Its goal is supervised learning: Given a dataset, each of the tuples can be described by a set of attribute values. each tuple belongs to a type in a mutually exclusive category. C4.5 aims to find a ing from the property value to the category through learning, and this ing can be used to classify entities with unknown new categories.

C4.5 was proposed by J. Ross Quinlan Based on ID3. The ID3 algorithm is used to construct decision trees. A decision tree is a tree structure similar to a flowchart. Each internal node (non-leaf node) represents a test on an attribute, and each branch represents a test output, each leaf node stores a class label. Once a decision tree is established, a path from the root node to the leaf node is tracked for a tuples without a given class label, and the leaf node stores the prediction of the tuples. The advantage of decision tree is that it does not require any domain knowledge or parameter settings, and is suitable for Knowledge Discovery of probing.

C4.5 and cart algorithms are derived from ID3 algorithms, which are important in data mining. Is the decision tree generated by a typical C4.5 algorithm on the dataset.

As shown in Dataset 1, it represents the relationship between weather conditions and whether to play golf.

Figure 1 dataset

Figure 2 decision tree generated by C4.5 on a dataset

Algorithm Description

C4.5 is not an algorithm, but a group of algorithms-C4.5, which are not pruning C4.5 and C4.5 rules. The algorithm in will provide the basic workflow of C4.5:

Figure 3 C4.5 algorithm flow

We may have doubts that a single tuples have many attributes. How do we know which attribute should be determined first, and which attribute should be determined next? In other words, in Figure 2, how do we know that the first property to be tested is outlook, not windy? In fact, one concept that can answer these questions is to select a measurement for the attribute.

Attribute selection Metric

Attribute selection metrics are also known as splitting rules because they determine how tuples on a given node are split. Attribute selection measure provides the rank evaluation for each attribute to describe the given training tuples. The attribute with the best measurement score is selected as the split attribute of the given tuples. Currently, popular attribute selection metrics include information gain, gain rate, and Gini.

First, let's make some assumptions. Set D to the training set of the class tag tuples. The class label attribute has m different values and m different class Ci (I = ,..., M), CID is the set of cidrds in D, | d | and | CID | the number of tuples in D and CID respectively.

(1) Information Gain

Information gain is actually used in the ID3 algorithm for Attribute selection measurement. It selects the attribute with the highest information gain as the splitting attribute of node n. This attribute minimizes the amount of information required for the tuples In the result division. The expected information required for the tuples in D is as follows:

(1)

Info (d) is also called entropy.

It is assumed that the tuples in D are divided by attribute a, and attribute a divides d into V different classes. After the division, the information required to obtain accurate classification is measured by the following formula:

(2)

Information gain is defined as the difference between the original information requirement (based on the class ratio only) and the new requirement (obtained after dividing a), that is

(3)

I think a lot of people think this place is not very easy to understand, so I have studied the description in the document and compared the above three formulas, let's talk about my own understanding.

Generally, it is almost impossible for a single attribute to completely separate a single attribute. Otherwise, the depth of the decision tree can only be 2. From this, we can see that once we select a property A, we assume that the component is divided into two parts, A1 and A2. Since A1 and A2 can also be divided by other attributes, so a new question arises: which attribute should we choose to classify next? The expected information required for the meta-group classification in D is Info (d). Similarly, when we divide d into V subsets by a (j = 1, 2 ,..., V) then, we need to classify the DJ's tuples. The expected information is Info (DJ), and a total of V classes, so we need to reclassify the V sets, the required information is formula (2. It can be seen that if formula (2) is smaller, does it mean that the smaller information we need to classify the several sets that A is split up next? In fact, Info (d) is fixed for a given training set, so the attribute with the maximum information gain is selected as the split point.

However, there is actually a disadvantage when using information gain, that is, it tends to have attributes with a large number of values. What does it mean? That is to say, in the training set, the more different values a certain attribute has, the more likely it will be used as a split attribute. For example, a training set has 10 tuples. For a certain Phase A, it takes the numbers 1-10 respectively. If a is split into 10 classes, for each class, Info (DJ) = 0, so formula (2) is 0. The information gain obtained by dividing this attribute (3) is the largest, but obviously, this division is meaningless.

(2) Information gain rate

Based on this, C4.5 following ID3 adopts the concept of information gain rate. The information gain rate uses the "split information" value to normalize the information gain. The classification information is similar to Info (D) and is defined as follows:

(4)

This value indicates the information generated by dividing the training dataset d into the V partitions output by the V partitions in the attribute a test. Information gain rate definition:

(5)

Select the attribute with the maximum gain rate as the split attribute.

Ii. Algorithm Description

(1) We want to construct a decision tree. Naturally, each layer of the tree represents an attribute value, and the final leaf node points to the classification class. 2.

(2) The natural problem is how to select appropriate nodes on each layer to construct the tree so that the structure of the tree is as optimal as possible, that is, the search path is as short as possible.

(3) The most critical issue is how to find the most suitable splitting node from the remaining unallocated nodes on each layer.

(4) the ID3 algorithm selects the optimal node by selecting the attribute with the highest information gain. Information gain can be simply understood as reducing uncertainty by a small amount after a certain attribute is used for Division.

(5) The C4.5 algorithm makes an improvement by using the attribute with the highest information gain rate. The advantage of doing so is that the tree width can be avoided.

(6) Some pruning operations should be performed after the tree is built. Of course, this operation is not reflected in the mainstream algorithm line, nor is it forced, but you can pay attention to how WEKA is implemented.

Iii. Main data structures used in algorithms

(1) instances object

An instances is a table that corresponds to an ARFF file or a CSV file. The instances object can be used to obtain the mean variance of a column. It is mainly an encapsulation of several Row Records.

(2) instance

One instance represents a record. In other words, the data of one instances contains multiple instances. Each instance has a special column classindex. The column value represents the type of the instance, specifically the golf in Figure 1.

(3) classifier Interface

In WEKA, each classifier inherits from this interface (although it is an interface in the sense, it is actually a subclass). This interface provides a buildclassifier method to pass in an instances object for training, the classifyinstance method is also used to pass in an instance to determine which class it belongs.

(4) j48

Classifier main class that implements the classifier interface.

(5) classifiertree Interface

Represents a node in the tree, maintaining and composing the tree structure. J48 uses c45pruneableclassifiertree and pruneableclassifiertree.

(6) modelselection Interface

This interface is used to determine and select the optimal attribute. Different instances are placed in different subsets Based on the attribute. The classifiertree interface uses modelselection to generate a tree structure. This abstract method is worth learning. The implementations of this interface used in j48 include binc45modelselection and c45modelselection, the first one is to generate a binary tree (that is, whether each node contains only two answers), and the other one is to generate a standard c45 tree.

Not complete ......

Source code analysis of WEKA algorithm Classifier-tree-J48 (I) algorithm and Basic Data Structure