Information Gain of Feature Selection Method

As mentioned above, in addition to Chi, information gain (IG) is also a very effective feature selection method. However, feature selection always quantifies the importance of features before selection. How to quantify the importance of features becomes the biggest difference among various methods. In the square test, the correlation between features and categories is used for quantification. The higher the correlation, the higher the feature score, and the more the feature should be retained.

In information gain, the importance measure is to see how much information a feature can bring to a classification system. The more information it brings, the more important the feature is.

Therefore, let's first review the definitions of Information in Information Theory (that is, "entropy. If there is such a variable X, it may take n or more values, each of which is x1, x2 ,......, XN, each of which has a probability of being P1, P2 ,......, Pn, then the entropy of X is defined:

It means that a variable may change more (instead, it has nothing to do with the detailed value of the variable, only the type of the value and the probability of occurrence ), it carries a larger amount of information (so I always think that our policies and regulations have a large amount of information, because it has changed a lot and basically changes and laughs ).

For the classification system, Category C is a variable, which may take C1, C2 ,......, CN, and the probability of each category is P (C1), P (C2 ),......, P (CN), so n is the total number of classes. At this time, the entropy of the classification system can be expressed:

Some people say it is hard to understand. Just think so. The function of the text classification system is to output a value indicating the category of the text, which may be C1, C2 ,......, CN, so the amount of information carried by this value is so much in the above formula.

Information gain is for one feature, that is, to look at a feature T. When the system has it and does not have it, the amount of information is different, the difference between the two is the information that this feature brings to the system, that is, the gain. When the system contains feature t, the amount of information is calculated very well, that is, the formula just now, which indicates the amount of information about the system when all features are included.

The problem is how to calculate the amount of information when the system does not include t? Let's think about the problem from another angle and think about what the system has to do as follows: There are many seats in the classroom, and every time the students come in, they can sit freely, as a result, the change is very large (countless possible seat situations); but now there is a seat that is very clear on the blackboard, and it is also very clear to the teacher, as a result, the headmaster's sister-in-law's daughter entrusted the relationship (it was really a move) and settled the seat. She could only sit down for her every time, but no one else could. What is the situation now? For the possible seat situations, we can easily see that the following two situations are equivalent: (1) There is no seat in the classroom; (2) Although there is a seat in the classroom, but others cannot sit down (because it cannot be changed, it remains unchanged ).

Correspondingly, our system is equivalent to the following: (1) the system does not include feature t; (2) Although the system includes Feature T, T is fixed and cannot be changed.

When the classification system does not include feature T, we use case (2) to replace it, that is, to calculate the amount of information in the system when a feature t cannot change. In fact, this information actually has a special name, called "Conditional Entropy". The condition naturally refers to the condition that "t has been fixed.

However, the problem arises one after another. For example, a feature X may have n or more values (x1, x2 ,......, XN) When Conditional Entropy needs to be fixed, which value should it be fixed on? The answer is that each of them should be fixed, and N values should be calculated, and then the mean value is the conditional entropy. While the average value is not simply added with a plus and then divided by N, but the probability of each value appearing is used to calculate the average (simply put, a value is more likely to appear than the limit, the proportion of the amount of information calculated on it is more ).

Therefore, there are two conditional entropy expressions:

This is the conditional entropy when feature X is fixed to the value Xi,

This refers to the conditional entropy when feature X is fixed. Note the difference between it and the above formula. We can see from the discussion of the average calculation just now that the relationship between the second formula and the first formula is:

What are the possible values of feature t in our text classification system? Note that T refers to a fixed feature. For example, it refers to the keyword "economy" or "Sports". When we say that the feature "economy" may have only two values, "economic" either appears or does not appear. Generally, the value of T is only T (indicating T appears) and (indicating t does not appear). Note that the system includes t, but t does not appear.

Therefore, the conditional entropy of the system is fixed when T is fixed. In order to distinguish the symbols of T and the symbols of feature t itself, t is used to represent the features, and T is used to represent T. So:

Compared with the previous formula, p (t) is the probability of T occurrence, that is, the probability of t not appearing. This statement can be further expanded.

Another half can be expanded as follows:

Therefore, the information gain that feature t brings to the system can be written into the difference between the original entropy of the system and the conditional entropy after fixed feature T:

There seem to be a lot of things in the formula, and they are actually very computation. For example, P (CI) indicates the probability of occurrence of class CI, in fact, you only need to divide by 1 by the total number of classes. (this means that you treat each category equally and ignore them in the hour, if we consider the size, we need to add the influence of the size ). For example, p (t) is the probability that feature T appears. simply divide the number of documents that have appeared t by the total number of documents, for example, P (CI | T) it indicates the probability that a class CI will appear when T appears. simply use the document count that appears T and belongs to the class CI divided by the document count that appears T.

From the above discussion, we can see that the information gain also takes into account both the appearance of features and the absence of two cases. Like the square test, the information gain is more comprehensive than the gain, and thus the effect is good. However, the biggest problem with information gain is that it can only examine the contribution of features to the entire system, rather than detail a specific category, this makes it only suitable for "Global" feature selection (that is, all classes use the same feature set ), however, it is impossible to select "local" features (each category has its own feature set. Due to some words, this category is very distinctive, and another category is insignificant ).

Let's see, the export process is actually very easy, and there is no magical right or incorrect. In some academic papers, I like to write such straightforward things very obscure, as if only some readers do not understand it is the true success of the author.

We are a new generation of scholars. We do not have the knowledge to be seen by others, and we do not have the knowledge to teach others. So we can make things simple and clear. Hello everyone, it's really good.

Information Gain of Feature Selection Method