Inverted index simple comprehension

http://blog.csdn.net/hguisu/article/details/7962350

http://blog.csdn.net/hguisu/article/details/7969757

===================== I'm a split line =============================

Inverted Index (English: Inverted index), also often referred to as a reverse index, place file , or reverse file, is an indexed method that is used to store the mapping of a word in a document or group of documents under a full-text search. It is the most commonly used data structure in the document retrieval system. By inverted index, you can quickly get a list of documents that contain the word based on the word. The inverted index consists mainly of two parts:the word dictionary and the inverted file.

Inverted indexes have two different types of reverse indexes:
A record's horizontal reverse index (or Reverse archive index) contains a list of documents for each reference word.
The horizontal reverse index of a word (or a full reverse index) also contains the position of each word in a document.
The latter form provides more compatibility (such as phrase search), but requires more time and space to create.
Indexes that are caused by modern search are based on inverted indexes. Compared with the index structure such as "signature file", "suffix tree","inverted index" is the best way to implement the relationship between word-to-document mapping and the most efficient index structure.

The data structure commonly used in Word dictionaries has loads hash linked list tree structure

1. Simple Index Construction

The establishment of indexes is equivalent to the process of establishing the table from the positive to the inverted list.

Process:

Word Rewind

2, the merger method to establish the index

The merge method, that is, each time the in-memory data is written to disk, all intermediate result information including the dictionary is written to the disk, so that all the contents of the memory can be emptied, and subsequent indexing can use all the fixed memory.

Merge process:

1) page analysis, generate temporary inverted data index A, B, when the temporary inverted data index A, B, fills the memory, the memory index, a, a temporary file is written to create a temporary inverted file,
2) to generate multiple temporary inverted files, perform the multi-merge, the output to get the final inverted files (inverted file).

Page analysis during index creation, especially Chinese participle is the main time overhead. The second step of the algorithm is relatively fast. The optimization of the creation algorithm concentrates on the efficiency of Chinese word segmentation.

3. Parallel and distributed indexing

Inverted index simple comprehension