General principles of Document Indexing

 

--Lvpei.cnblogs.com

Full-text retrieval is widely used by search engines, and the large-scale documents processed by them are non-structured or semi-structured information. Document indexing is the basis for full-text retrieval and quick query of information.

I. Index Overview

An index is an organizational structure of data storage. The indexer is used to create, maintain, and manage information indexes.

Inverted index is an efficient index organization method. It uses words or words as index items. It can support multiple search models and provide high-performance search. The search engine index uses the index model of traditional information retrieval. The index structure is usually organized into a linked list arranged by index items. During retrieval, you can use the search term to match the index item and locate it directly in the list where the search result is located.

 

Ii. Enable and disable the environment of the Indexer

The indexer extracts index items from the original document to represent the document and generates an index table for the document set. The indexing environment involves text preprocessing and database retrieval. It mainly includes the original text Library (webpage or document) and text Parser (analysis webpage and document content, unified as a non-format document or intermediate format document ), text intermediate format Library (remove noise junk information, pure text after word segmentation or unified intermediate format), indexer (process the result information processed by the parser, use the preset index item dictionary to create a linked list arranged by index items, and dynamically change the index dictionary), index Dictionary (Word and encoding Binary Group structure database ), text Index Library (organizes data files stored in the form of indexes ).

 

Iii. inverted index Principle

The core mechanism of indexing technology is the inverted index structure. Large-scale data is generally stored in the form of files. The inverted index uses the index keyword to directly identify the document list and finally determine the document to be found.

 

Iv. Index classification

Index classification generally involves two aspects: the types of index items processed by indexes (word index, word index, phrase index, and hybrid index) and index creation Update Form (static and dynamic ).

 

V. High-Performance Indexing

1-how to increase the speed of indexing.

2-how to reduce the resource usage of indexes.

3-how to reasonably allocate limited memory resources during index usage.

4-how to speed up resource access.