Small experiment of Chinese word segmentation based on Internet content

Word segmentation is useful for search engines, input prompts, speech recognition and other human-computer interaction systems. The huge amount of information on the Internet provides a large number of dynamic updates of the original database for Word segmentation research. Many internet companies have word breakers based on Internet content. and using Hadoop to analyze large amounts of data. These sounds big, but it's easier to think about the fundamentals, and it's easy for every programmer to look into the field. So a simple POC experiment was done.

Start by writing a spider that crawls content from the Internet page. Because it's just a simple experiment, I crawl only one page, not the recursive crawl link page. Crawl content filters out HTML tags leaving plain text.

The second part, write a continuous frequency analysis between a word analyzer. The basic idea is to define a read character buffer, read the stream of input characters, to each word x:

1. If the separator character is a space, the cache is emptied.

2. If it is a normal character: if the cache already has content E then record <e, x> match number plus one. The x is then also put into the cache. (Problem: You should have cardinality plus one for any subsequent substring of E.) ）

In this way, we can finally draw a list of possible characters followed by each string and the corresponding number of matches. In this experiment, I used the file system directory structure to record: each string a folder, and then may appear after each word a file, the file is empty, with the file's lastmodified timestamp records the number of matches. This is not feasible in a real system and must be designed to store data structures, but for the sake of simplicity, we use the file system in this experiment.

The third part is to do a Java swing interface, give an input box, the user input any string, and then based on user input to find the most common match of the next character. Here are a few questions:

1. Find only one character is not enough, such as user input "Chinese people", we should give "republic" rather than just hint a "total". So after matching one, we need to give the next matching character on this basis.

2. If the user input cannot be matched to the next character, the leftmost character of the user input string should be removed, and the match will continue according to the right string, recursively until a match is found or the string length is zero. (Because we record all possible, in fact, when the probability is less than a certain value should be considered no longer the future word.) ）

3. The above two situations can not be confused when considering together, for example, can not match a continuous but meaningless word solitaire, such as "Chinese family militia seed", so match the next word to consider the entire previous participle, or continue to find the next possible character. The situation here is slightly complicated. In this simple experiment it was just a simple process.

In general, this small experiment gives a more meaningful result. Of course, in a real product to do more research on the relevant issues.

The code is here: Http://blog.luoxq.com/files/fenci.tar altogether only 200 + lines of code.

The actual results of the interface are as follows:

Small experiment of Chinese word segmentation based on Internet content