55.TF/IDF algorithm

Key points of knowledge:

• TF/IDF Algorithm Introduction
• View es Calculation _source the process and the score of each entry
• View a Document how it was matched to the

First, the algorithm introduction

Relevance Score The algorithm, in a nutshell, is to calculate the degree to which the text in an index matches the search text, and the correlation between them. Elasticsearch uses the term frequency/inverse document frequency algorithm, referred to as tf/ IDF Algorithm

1.Term Frequency

each entry in the search text is Field How many occurrences of the text, the more occurrences , the more relevant

Example

Search Request: Hello World

Doc1 : Hello, and world is very good

doc2 : Hello, how is it

Doc1 more relevant than DOC2

2.Inverse Document Frequency

The more occurrences of each entry in the search text in all documents throughout the index, the more irrelevant it is.

Search Request: Hello World

Doc1 : Hello, today is very good

doc2 : Hi World.

For example, inIndexin a1million BarDocument,HelloThis word is in all theDocument, there was a total of +times; WorldThis word is in all theDocument, there was a total of -times, thendoc2thanDoc1more relevant

3,field-length norm

The longer the field, the weaker the correlation

Search Request: Hello World

Doc1 : {"title": "Hello article", "Content": "Babaaba 1 words "}

doc2 : {"title": "My article", "Content": "Blablabala 1 words,hi World"}

Hello World in the entire Index is the same number of occurrences, but Word is only present in 10,000 words, so Doc1 more relevant, Title Field even shorter

Second, see How the _score is calculated

Get/test_index/test_type/_search?explain

{

"Query": {

"Match": {

"Test_field": "Test Hello"

}

}

}

third, see how a document is matched to

Get/test_index/test_type/6/_explain

{

"Query": {

"Match": {

"Test_field": "Test Hello"

}

}

}

Iv. Extension of Reading

Baidu Encyclopedia

 

55.TF/IDF algorithm