Article Classification of Big Data

Write it down or forget it later.

　　

Analysisentry: The overall transfer, the order of the class;
Wordfrequenceindoc: Extract Chinese, participle, to stop the word, statistical frequency;
To stop the word, do a thesaurus, my.dic or Stopword.dic
Wordcountsindoc: Count the number of words in each document;
WORDSINCORPUSTFIDF: Count words in how many documents appear, calculate TFIDF, set up a glossary;

SORTTFIDF: To sort the TFIDF;

Combinationkey, a class is composed of a key (two fields or multiple fields that are combined into a key), sorted by key, and called in SORTTFIDF;
Definedcomparator defines a comparer, which is called when sorting to compare;
Definedpartition defines the partition, Partitioner is the role of the mappper produced by the intermediate results of sharding, so that the same packet of data to the same reducer processing, it directly affects the reduce phase of load balancing. The definepartition defined here is to fragment by key;

Docvetorbuild: Creating a word vector this is because when using the SVM algorithm, the input is using a vector
USESVM: Call SVM algorithm to classify;
SVM belongs to supervised class algorithm, it needs to make data as two parts, part as training set, part as test set (i.e.,
The first part of the data, as a standard training, and the new data, the time to test (prediction);

Test.java is used to test some functions, after testing good, in use;

Article Classification of Big Data