Bayesian text classification experiment

This is actually a big job of Data Mining:

Experiment Principle

It mainly uses Naive Bayes to classify the text, and uses TF-IDF weight for optimization.

Experiment process: 1 first download sogou Corpus

Five of these categories are used: 1-vehicle 2-finance 3-education it4-Health 5-sports. There are 1000 training documents and 131 test documents for each classification.

2. Chinese Word Segmentation

The Chinese Emy of Sciences word segmentation software is used to separate all documents. For example, the result of 10.txt in the first category is stored in javasre_10.txt. Word splitting result (only part of the list ):

/X newspaper/Rz reporter/n Chen Xue/NR Frequency/Ag internship/V reporter/n Tang Xiang/NR from/V Shanghai/NS

/X A/n just/D was established/VI two years/M/ude1 Network/n payment/VN Company/n, /WD it/RR/ude1 target/N is/vshi become/V Market Value/N 10 billion/m usd/Q/ude1 listed/VN Company/n. /WJ

His results can be followed by the part of speech. When we create a dictionary, we only use nouns.

Then all the training documents and test documents are stored in the same directory. My program reads an ls. txt file to decide which files to process. I use 1000-1999 for each category as the training document and 10-140 as the test document.

For the first classification, both 1_re_10 and 1_re_140 are test documents, and 1_re_1000 to 1_re_1999 are training documents.

3. Create a dictionary

Conduct Word Frequency Statistics on all documents and only record words higher than a certain times. Then, use stopword.txt to remove some deprecated words. In this way, a dict.txt file is obtained. The format of dict.txt is: Word space word frequency. The dictionary contains 33845 nouns.

In this example, you can generate dict.txt. You can manually judge that some words are used as the new deprecated words and put them in stopword. Then run the program again and the new dict.txt will be generated next time.

4. Use Naive Bayes algorithm to classify texts

Bayes algorithm: Assume that D is an arbitrary document, and document D is represented by the feature words it contains, that is, D = (T

1, T2 ,......, TK). k indicates the number of feature words in the document. TJ indicates the J-th Feature Word in the document D. The following formula is obtained:

;
Indicates that the feature value of classifier prediction is in the category
The probability of occurrence in CJ's document.

According to Bayesian theorem:

Because P (d) is a constant for all classes, and there is a reason for C ++ floating point operations, there is no need to multiply the sum proportional
.

Indicates CJ Feature Word Ti
The probability of occurrence is equal to the occurrence frequency of the Feature Word Ti divided by the total number of words in CJ classification.

See the tf-idf-nb.cpp: Bayes () is used to train the training text, int GetType (string filename) returns the type of the file. The following describes the main code area.

/* Calculate P (Ti | CJ) count (Ti)/count (CJ ). The probability of occurrence is divided by the total number of words in the classification. Because the number of documents obtained by my respective categories is certain, my P (CJ) = 1/N */ For (type = 0; Type <n; Type ++) { Long sizeint = bmap [type]. Size (); Map <string, double >:: iterator ITER; For (iter = bmap [type]. Begin (); iter! = Bmap [type]. End (); ITER ++) { ITER-> second = (ITER-> second + 1.0)/(count_classify [type]); // here is the calculation } }

While (input> word) { String rc = word. substr (0, word. Find ("/")); // Cout <RC <Endl; For (I = 0; I <n; I ++)/* This is the probability that the test document is accumulated under each category */ { Iter = bmap [I]. Find (RC ); If (ITER! = Bmap [I]. End ()) Result [I] + = ITER-> second; } }

However, the accuracy of the word frequency method is only 73%.

5 Optimization Using TF-IDF

The naive Bayes used previously uses the word frequency weight, which is highly dependent on text preprocessing. Then I searched for improvement methods online and used TF-IDF as the new weight calculation method.

In computing
, And then multiply the weight. The IDF weight.

N indicates the total number of documents, and N (TK) indicates the number of documents containing the word TK. The greater the IDF value, the less documents that contain the feature word. If the Feature Word is distributed in a concentrated manner, the Feature Word may contain more category information. On the contrary, if the IDF value is small, it indicates that the Feature Word is evenly distributed in the document set and has little contribution to the document classification. This word is not suitable for classification features.

Modify the following table in the Code:

While (input> word) { String rc = word. substr (0, word. Find ("/")); // Cout <RC <Endl; For (INT I = 0; I <n; I ++) { Iter = bmap [I]. Find (RC ); If (ITER! = Bmap [I]. End () & dictmap. Find (RC )! = Dictmap. End ()) { If (weightmap. Find (RC )! = Weightmap. End ()) Result [I] + = (ITER-> second * (log (documentn/(weightmap. Find (RC)-> second )))); Else Cout <"can not find" <RC <"in weightmap" <Endl; } } }

The experiment result is:

Here, I use numbers to represent a category. There are five categories in total. 5 represents 5th categories of documents. Then, I use the improved algorithm to calculate the categories. The following describes the accuracy rate of each classification and the overall accuracy rate.

Original blog address