Lucene source code analysis-11 how to analyze the document content

Source: Internet
Author: User

Reprinted from

The source address is a Word document, which is converted to HTML format.


LuceneSource code analysis

How is the document content analyzed?


AnalyzerClass is responsible for analyzing the document structure and extracting content.


6.1Document AnalysisAnalyzer

6.1.1Org. Apache. Lucene. Store. Analyzer

Analyzer Class Construction for Text Analysis Tokenstream Object, therefore ( Thus ) It indicates ( Represent ) Is used to break down from text ( Extract ) Terms A rule server ( Policy ). Typical ( Typical ) First, create Tokenizer , It will Reader Object To read the upstream stream ( Stream of Characters ) ( Break ) Original Tokens ( Raw tokens ). Then one or more Tokenfilters Can be applied in this Tokenizer . Warning: you must ( Subclass ) Override ( Override ) Defines one of the methods in this class. Otherwise Analyzer Will enter an infinite loop ( Infinite Loop .


6.1.2Org. Apache. Lucene. Store. standardanalyzer

Standardanalyzer Class is to use English Of Stop Words List Tokenize Split into text Word , Use Standardtokenizer Class decomposition word, plus Standardfilter And Lowercasefilter And Stopfilter These filters process such Analyzer Class implementation.



Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.