Introduction to Information Retrieval Study Notes (2) word term dictionary and inverted record table

Introduction to Information Retrieval Study Notes (2) word term dictionary and inverted record table

Document Analysis and encoding conversion

1. Generate character sequences

2. Document Unit selection

Word set selection

(1) Entry-based

Word-based processing is often related to the province of the language, and the word-based processing in different languages is different.

(2) Remove deprecated words

(3) Merge Processing

(4) stem reduction (stemming) and lemmatization)

Fast merge of inverted record tables based on table jumpsAlgorithm(Skip List)

Insert a jumper at the position of P, which is only used for A and B.

Inverted Sequence Table with location information