99.99% accuracy of news text classifier based on Chinese character word frequency feature (ii)

Realization of 99.99% accuracy rate based on Chinese character word frequency feature news text classifier (i)

99.99% accuracy of news text classifier based on Chinese character word frequency feature (ii)

99.99% accuracy of news text classifier based on Chinese character word frequency feature (iii)

Review

In the previous section, you have downloaded the extracted 80,000 text files in the Classfile directory of the D drive.

In this section, the word frequency features are extracted and categorized. We will see the legendary 99.xx% ...

Source

Click Download to generate the source code for the word frequency feature data file: Predata.zip (15.57 KB)

A detailed description of this source code will be explained in a subsequent article. Now, let's take a quick look at the legendary 99...%.

First, unzip the source code, and then open the PreData.sln project file

When vs loads the project (if the computer is an older version of VS and does not have the. NET Framework 4.5, click Modify to. NET Framework 4.0), make sure that D disk has enough free space (around 700M) to click the triangle to compile the code, and run the Chinese character Word frequency feature extraction program.

The program will load the text data in the D-disk Classfile directory for two scans. and generate the feature data.

Exciting first time

Exciting time to come, with the data, you can ...

Let's start by using the Svm-train.exe tool in the Popular LIBSVM Toolkit (downloaded from the LIBSVM website) to see what surprises this feature data can bring. As shown, enter the command svm-train.exe-v 4-c data.txt

Parameter meaning: Divide 80,000 characteristic data into four parts, cross verification, penalty coefficient 1000.

After a long wait (about one hours on the author's computer), we got:

Well, this result, but the traditional word frequency method is difficult to achieve the characteristics. Moreover, it is rough to get on the premise that the data is not being cleaned with complex data.

Well, I admit, this data and 99.xx% still have 0.xx% distance. Then let's take a small step forward.

next section Trailer

Well, in this section, we generate a feature dataset based on the word frequency of Chinese characters, and, with nothing extra, we use SVM's training tool to get a model close to 99%.

In the next section, we will be offering a five-layer artificial neural network model. He will bring ....

It's too late to rest first.

99.99% accuracy of news text classifier based on Chinese character word frequency feature (iii)

99.99% accuracy of news text classifier based on Chinese character word frequency feature (ii)