Lucene source code analysis-11 how to analyze the document content

Reprinted from http://download.csdn.net/source/858994

The source address is a Word document, which is converted to HTML format.

 

LuceneSource code analysis

How is the document content analyzed?

 

AnalyzerClass is responsible for analyzing the document structure and extracting content.

 

6.1Document AnalysisAnalyzer

6.1.1Org. Apache. Lucene. Store. Analyzer

Analyzer Class Construction for Text Analysis Tokenstream Object, therefore ( Thus ) It indicates ( Represent ) Is used to break down from text ( Extract ) Terms A rule server ( Policy ). Typical ( Typical ) First, create Tokenizer , It will Reader Object To read the upstream stream ( Stream of Characters ) ( Break ) Original Tokens ( Raw tokens ). Then one or more Tokenfilters Can be applied in this Tokenizer . Warning: you must ( Subclass ) Override ( Override ) Defines one of the methods in this class. Otherwise Analyzer Will enter an infinite loop ( Infinite Loop .

 

6.1.2Org. Apache. Lucene. Store. standardanalyzer

Standardanalyzer Class is to use English Of Stop Words List Tokenize Split into text Word , Use Standardtokenizer Class decomposition word, plus Standardfilter And Lowercasefilter And Stopfilter These filters process such Analyzer Class implementation.