SOLR Chinese search inverted index and Data Storage Structure

As a search, our traditional method (positive index) starts from the key point, and then finds the specific information that can meet the search conditions in the Key Point information, both search for value through key. Lucene uses inverted indexes to search for keys. In Chinese full-text search, value is the word to be searched, and the place where all words are stored is called a dictionary. Key is the document label list. (You can find the document that you want to search for the word value in the document label list ).The forward index finds words from the document number:

The inverted index is used to find the document number from the word:

When the document data comes, SOLR first splits the document data and creates an index database and a document database. Word segmentation refers to dividing a text segment into several words according to certain rules. For example, how to generate word segmentation storage after SOLR is used in the following two documents:

Punctuation Marks in the article can be filtered out directly, such as and too can be filtered out directly. Word Segmentation table representation:

Lucene's inverted index storage structure is:String of word items + Document Frequency of word items + frequency of recording word items + location information of recorded word items + jump offset. A simple understanding can form the following structure:

Indicates the document number of the word, the frequency of the word in the document, and the position of the word in the document. In this way, when we search for a word, we will find the IDs of all the documents that appear in the word, and then find the specific content of the document through the Document ID.