Basic concepts of search engines (3) -- inverted list

Inverted list

The inverted list is used to record which documents contain a word. Generally, there are many documents in the document set that contain a word. Each document records the document number (docid) and the number of times the word appears in this document (TF) and where the word appears in the document, so that the information related to a document is called as an inverted index (posting ), A series of inverted indexes containing this word form a list structure, which is the inverted list corresponding to a word. Figure 1-1 is an inverted list. All words that appear in the document set and their corresponding inverted lists constitute an inverted index.

Figure 1-1 inverted list

In the actual search engine system, instead of storing the actual document numbers in inverted index items, the actual document numbers are replaced by the document number difference value (D-gap ). The document number difference is the difference between the document numbers of two adjacent inverted index items in the inverted table, the document number displayed after the inverted table is greater than the document number displayed before. Therefore, the document number difference is always an integer greater than 0. In the example from 1 to 10, the numbers of the original three documents are 187, 196, and 199 respectively. By calculating the number difference, it is converted: 187, 9, 3.

Figure 1-2 Sample document No. Difference

The main reason for calculating the difference between document numbers is to better compress the data. The numbers of the original documents are generally large values, this effectively converts a large value to a small value, which helps increase the data compression rate.

 

Basic concepts of search engines (3) -- inverted list