Research on TFIDF algorithm based on MapReduce programming model

Research on TFIDF algorithm based on MapReduce programming model

Zhao Weiyan Wang Jingyu

With the rapid development of technology such as Internet, information processing has become an indispensable tool for people to obtain useful information, and how to obtain useful information efficiently in mass information is very important. The existing text classification algorithm meets the bottleneck in time complexity and space complexity, and it can not meet the needs of people, therefore, a TFIDF algorithm based on Hadoop distributed platform is proposed, the specific flow of the algorithm is given, and the algorithm is realized by MapReduce programming, Compared with the traditional serial algorithm, the comparison experiment is carried out in single machine and cluster mode. Experiments show that the TFIDF text classification algorithm can be used to efficiently classify massive data.

Research on TFIDF algorithm based on MapReduce programming model