TF: A correlation ranking technique for traditional IR (II.)

Intermediary transaction http://www.aliyun.com/zixun/aggregation/6858.html ">seo diagnose Taobao guest cloud host technology Hall

Since it is two, it is followed by the previous article "TF: Traditional IR related sorting technology" written down. Therefore, interested students please read the first article before you continue.

Okay, let's move on to the second.

On the word frequency, as long as you have the tool of participle good enough to understand and realize. About the anti-document frequency, dear friends, when you see the beginning of the feeling is very bull fork, and then thin think will be very confused?

Reverse document Frequency (IDF) =log (total number of documents/number of documents containing keywords)

Yes, the puzzle is how to get the total number of documents and the number of documents containing keywords.

On the search engine, can have a good alternative way, listen to me carefully below.

Almost every page of each article contains the word "the", well, you think of it. Search engine in the word, the number of results can be understood as the number of documents, and then search your target word is the number of documents containing the word, this data is resolved, the following is an example of me:

Well, with this data, let's see what we can do.

Each page in the Web site to the word, remove the modal particles after the words in accordance with the TF value from large to small to sort.

Web A={a1,b1,c1,d1,e1......z1}

Web B={a1,b2,c1,d5,e2......z6}

Web C={A2,B1,C2,D1,E2......Z2}

......

It is clear from {a1,b1,c1,d1,e1......z1} that the meaning of page A is understood, as are B and C.

If the words in a, B, C are compared by a method, then it is not possible to calculate ..., you think right, the similarity between pages.

This method is the cosine. Specific actions, as follows:

We first select from A, B, C, the top n can express the theme of the page to form a collection.

{A1,C1,D1,E1,B2,D5,E2,A2,B1,C2}

Then calculate a, B, C page for each word in the set frequency of words (if necessary, please use relative frequency), composition of the corresponding vector.

a=[2,1,3,5,0,0,0,0,1,0]

B=[...]

C=[...]

Remember the formula you learned in high school.

OK, after the calculation of this formula, not only is the similarity between the pages, the same page of the most relevant recommended articles can also be generated.

Interested students, please test it.

Reprint please indicate the link address http://www.seosos.cn/search-engine/tf-idf-application.html.