Lucene word segmentation-LetterTokenizer

There are many articles about word segmentation in the garden. birdshover wrote some articles about word segmentation. here, I mainly go deep into the internal algorithms of the Lucene word splitting tool and hope to communicate with you.

The structure of Lucene class related to word segmentation is as follows:

(Picture taken from: http://www.cnblogs.com/birdshover/archive/2008/08/28/1279044.html)
This section mainly discusses LetterTokenizer and CharTokenizer. In fact, the implementation of algorithms is implemented in next () in CharTokenizer.

First, let's look at a LetterTokenizer example to have a perceptual knowledge:

Code
Using System;
Using System. Collections. Generic;
Using System. Text;

Using Lucene. Net. Analysis;
Using System. IO;
Namespace ConsoleTestProject
{
Public class TestLetterTokenizer
{
// Test Data
Static string TEST_TEXT = "DM Database, DM Database! ";

Public static void Test ()
{
TokenStream ts = new LetterTokenizer (new StringReader (TEST_TEXT ));
Token token;
While (token = ts. Next ())! = Null)
{
Console. WriteLine (token. TermText ());
}
Ts. Close ();
}
Static void Main (string [] args)
{
Test ();
Console. Read ();
}
}
}

The output result is as follows:

DM Database

DM

Database

 

The next () method in LetterTokenizer inherits from CharTokenizer.

The next () method of CharTokenizer will be analyzed in detail below:

 

Code
Using System;

Namespace Lucene. Net. Analysis
{
/// <Summary> An abstract base class for simple, character-oriented tokenizers. </summary>
Public abstract class CharTokenizer: Tokenizer
{
Public CharTokenizer (System. IO. TextReader input): base (input)
{
}

Private int offset = 0, bufferIndex = 0, dataLen = 0;
Private const int MAX_WORD_LEN = 255; // maximum allowed Word Length
Private const int IO_BUFFER_SIZE = 1024; // maximum number of characters allowed at a time
Private char [] buffer = new char [MAX_WORD_LEN]; // The Word buffer to construct the Token data source.
Private char [] ioBuffer = new char [IO_BUFFER_SIZE]; // buffer for storing input strings

// Check whether the current character is a character that can generate a Token and leave it to the subclass for implementation
Protected internal abstract bool IsTokenChar (char c );

Protected internal virtual char Normalize (char c)
{
Return c;
}

/// <Summary> Returns the next token in the stream, or null at EOS. </summary>
Public override Token Next ()
{
Int length = 0; // Word length
/**
* Offset records the position of the original string in the current character, which is different from the bufferIndex value,
* BufferIndex indicates the position of the current character in the string read this time (its maximum value is limited by the ioBuffer size)
*/
Int start = offset;
While (true)
{
Char c;

Offset ++;
If (bufferIndex> = dataLen) // dataLen indicates the length of each read string
{
DataLen = input. Read (System. Char []) ioBuffer, 0, ioBuffer. Length );
BufferIndex = 0;
}
// Exit the loop if no data exists
If (dataLen <= 0)
{
If (length> 0)
Break;
Else
Return null;
}
Else
C = ioBuffer [bufferIndex ++]; // The bufferIndex is the displacement of ioBuffer characters.

If (IsTokenChar (c ))
{
// If it's a token char
If (length = 0)
// Start of token
Start = offset-1; // start indicates the start position of each word in the original string.

Buffer [length ++] = Normalize (c); // buffer it, normalized
// If the length of a word exceeds the maximum length, exit the loop.
If (length = MAX_WORD_LEN)
// Buffer overflow!
Break;
}
Else if (length> 0)
// At non-Letter w/chars
Break; // return 'em
}
// Encapsulate the data into a Token to return
Return new Token (new System. String (buffer, 0, length), start, start + length );
}
}
}

 

According to the comments in the program, it is easy to understand the general idea of the algorithm:

Traverses the input string, divides the input string into words according to the special symbol, and encapsulates the input string into tokens for return. The time complexity is O (n ).