TF-IDF algorithm Improvement

concept : TF-IDF (term frequency–inverse document frequency) is a commonly used weighted technique for information retrieval and information mining. TF-IDF is a statistical method used to evaluate the importance of a word to one of the files in a set of files or a corpus. The importance of a word increases in proportion to the number of times it appears in the file, but it decreases inversely as it appears in the corpus.

disadvantage : simply by "word frequency" to measure the importance of a term, not comprehensive enough, sometimes important words may not appear many times. Moreover, this algorithm can not reflect the position of the word, the occurrence of the position of the word and the occurrence of the post-position of the word, are considered to be the same importance, this is not true. Also did not consider part of speech.

idea : An article is generally composed of the title, the text, the most important part is often the title, the beginning of the article, the end of the article, and the text is composed of sentences. Generally speaking, the most important part of a sentence is the main predicate, and the subject and object is generally a noun, predicate is generally a verb, of course, adjectives can be modified nouns, adverbs can modify verbs.

program : You can give the title, the beginning of the article, the word at the end of the higher weight, calculate the keyword weight, you can consider the part of speech, generally speaking nouns > verbs > Adjectives, adverbs > Other