Full-text search based on e.net by Baidu counterfeited (1)

Lucene. Net

Lucene.net is from Lucene. net is an open source full-text search engine development kit, that is, it is not a complete full-text search engine, but a full-text search engine architecture, is a Library. you can also think of it as a simple and easy-to-use API that encapsulates the index and search functions (providing a complete query engine and index engine ). With this set of APIS, you can do a lot of searching, and it is very convenient .. Developers can implement full-text search based on e.net.

Note: Lucene. Net can only search text information. If it is not text information, convert it to text information. For example, to retrieve an Excel file, use NPOI to read the Excel file as a string and then throw the string to Lucene. Net. Lucene. Net saves the text that is thrown to it to speed up retrieval.

For more conceptual knowledge, refer to this blog: http://blog.csdn.net/xiucool/archive/2008/11/28/3397182.aspx

This Demo example shows:

OK. Next I will explain how Corporal can achieve this effect step by step.

Lucene. Net core-word segmentation algorithm Analyzer)

Learning Lucune. Net, word segmentation is the core. Of course, you can expand word segmentation on your own, but this requires high algorithm requirements. Different Word Segmentation Algorithms in Lucene. Net are different classes. All Word Segmentation Algorithms inherit from the Analyzer class. Different word segmentation algorithms have different advantages and disadvantages.

The built-in StandardAnalyzer is used to split English words by spaces, punctuation marks, and so on. The Chinese words are segmented by a single word. One Chinese character is counted as one word.

 

  
Analyzer analyzer = new StandardAnalyzer ();
  
TokenStream tokenStream = analyzer. TokenStream ("", new StringReader ("Hello Lucene. Net, I love 1 You China "));
  
Lucene. Net. Analysis. Token token = null;
  
While (token = tokenStream. Next ())! = Null)
  
{
  
Console. WriteLine (token. TermText ());
  
}
 

Result After word segmentation:

The binary word segmentation algorithm calculates one word for every two Chinese characters. "I love you China" is segmented into "I Love You china". Click to view the binary word segmentation algorithm CJKAnalyzer.

 

  
Analyzer analyzer = new CJKAnalyzer ();
  
TokenStream tokenStream = analyzer. TokenStream ("", new StringReader ("I Love You China People's Republic of China "));
  
Lucene. Net. Analysis. Token token = null;
  
While (token = tokenStream. Next ())! = Null)
  
{
  
Response. Write (token. TermText () + "<br/> ");
  
}
 

At this time, you must be thinking that there is no good binary word splitting algorithm, and you really want to expand Analyzer on your own, but it is not an algorithm professional. What should I do?

The fall of heaven, pangu word segmentation,Click to download.