TFIDF algorithm principle

TF-IDF (term frequency–inverse document frequency) is a commonly used weighted technique for information retrieval and information mining.

The main idea of TFIDF is that if a word or phrase appears in an article with a high frequency of TF and is seldom seen in other articles, it is considered to be a good category-distinguishing ability and suitable for classification.

TFIDF is actually: TF * IDF,TF Word frequency (term Frequency), IDF Anti-document frequencies (inverse document Frequency).

TF represents the frequency at which the entry appears in document D. The main idea of IDF is that if the fewer documents that contain the entry T, that is, the smaller the n, the larger the IDF, the better the class-distinguishing ability of the term T.

If the number of documents containing the entry T in a class of document C is M, and the total number of documents containing T in the other class is K, it is clear that all documents containing T are n=m + K, and when M is large, n is also large, and the value of IDF obtained by the IDF formula is small, indicating that the term T category is not strong.

In fact, if an entry is frequently present in a document of a class, it indicates that the term is a good representation of the character of the text of the class, which should be given a higher weight and selected as the characteristic word of the text to distinguish it from other classes of documents. This is where the IDF is deficient.

TF formula:

The above is the number of occurrences of the word in a file, and the denominator is the sum of the occurrences of all the words in the file.

IDF formula:

• | d|: Total number of files in corpus
• : The number of files that contain words (that is, the number of files) if the term is not in the corpus, it causes the divisor to be zero, so it is generally used

IDF = log (total number of files in corpus/number of files containing word t)

Then TFIDF is TF*IDF.

TFIDF algorithm principle