The use of Stanford-segmenter

For the Chinese Natural Language processing task, participle seems to be the first step, because the depth of learning to use the word vector, participle is basically the first step, unless we are based on a single Chinese word, of course, this look at the specific model. So we need a good word segmentation tool, stanford-segmenter is a good word breaker tool, of course, there are many excellent other word segmentation tools, specifically with which to see personal hobbies, this article will introduce the use of stanford-segmenter.

The toolkit I use is "stanford-segmenter-2015-01-30".

Like Postagger, it is written in Java, there are two ways to use it: Java; command line.

Java:

The specific code see Segdemo.java, also need to put the data folder under the project path, the build Add Stanford-segmenter-3.5.1.jar to the path, and eventually, if you need to generate a visual interface, you need to copy the data back under the generated executable jar path.

Command line:

Word of the command line execution mode can also see readme-chinese.txt this file, I use the segment.sh this, which, CTB or PKU optional.
I use segment.sh on the Linux server is relatively smooth, unlike postagger.sh also to change the Java classpath, so, direct use should be no problem, redirect the results can be divided into the results of the word.

segment.sh CTB Test.simp.utf8 UTF-8

Above ~