Maximum Matching segmentation algorithm

There are two important processes for full-text retrieval:

1 participle

2 Inverted Index

Let's first look at the word segmentation algorithm

At present, there are two directions for Chinese participle, one of which is the use of probability of the idea of the article participle. That is, if two words, the frequency of the occurrence is very high, we can assume that the two words are a word. Here can be measured in a formula: M (A, b) =p (AB)/P (a) p (B), where A is a word, B is a word, P (AB) indicates the probability that AB appears adjacent, p (a) indicates the frequency of A in this article, and P (b) indicates the frequency of B in this article. The advantage of using probability participle is that it does not need the help of dictionaries, the disadvantage is that the algorithm is troublesome, the efficiency is not high, and there is a certain error rate.
Another direction is to use dictionary participle. is to pre-prepare a dictionary for the program, and then through this dictionary to the article participle. At present, there are forward maximum matching algorithm and inverse maximum matching algorithm in the more popular ways. The inverse maximum matching algorithm is better in accuracy.

Take "I am a bad man" as an example, and the maximum word length is 3

The order of the Forward is
I was a
I am
I ===> get a word
is a
is a
is ===> get a word
A bad
A ===> get a word
Bad guy ===> get a word
As a result, I, a, a bad guy

Inverse algorithm
A bad guy.
Bad guys ==> bad guys.
is a
A ==> A
I am
Yes, ==> is.
I ==> me.
As a result, I, a, a bad guy

The Java code is as follows

Package Data;import java.util.arrays;import java.util.hashset;import java.util.set;/** * Maximum matching segmentation algorithm * * @author JYC506 * * /public class Splitstring {private set<string> Set = new hashset<string> ();p rivate int positiveover = 0;privat e int reverseover = 0;/** * Positive maximum match * * @param str to participle sentence * @param num Word maximum length * @return */public string[] Positivesplit (St Ring str, int maxSize) {int tem = 0;int length = Str.length (); string[] ss = new string[length];char[] cc = Str.tochararray (); for (int i = 0; i < length; i++) {positiveover = 0; String SB = This.tostr (CC, I, maxSize); ss[tem++] = Sb;i = i + positiveover;} string[] SS2 = new String[tem]; System.arraycopy (ss, 0, SS2, 0, TEM); return SS2;} /** * Add Thesaurus * * @param words */public void Addword (string[] words) {for (String st:words) {this.set.add (ST);}} /** * Inverse Maximum match * * @param str * @param num * @return */public string[] Reversesplit (String str, int num) {int tem = 0;int L Ength = Str.length (); string[] ss = new string[length];char[] cc = Str.tochararray (); for (int i = Str.length ()-1; i >-1; i--) {reverseover = 0; String SB = THIS.TOSTR2 (CC, I, num); Tem++;ss[--length] = Sb;i = I-reverseover;} string[] SS2 = new String[tem]; System.arraycopy (SS, Str.length ()-tem, SS2, 0, TEM); return SS2;}  Private String TOSTR (char[] cs, int start, int num) {int num2 = num;out:for (int j = 0; j < Num; J + +) {StringBuffer SB = new StringBuffer (); for (int i = 0; i < num2; i++) {if (start + i < cs.length) {sb.append (Cs[start + i]);} else {n Um2--;j--;continue out;}} if (Set.contains (sb.tostring ())) {Positiveover = Num2-1;return sb.tostring ();} num2--;} Return string.valueof (Cs[start]);} Private String TOSTR2 (char[] cs, int start, int num) {int num2 = num;for (int j = 0; j < Num; J + +) {StringBuffer SB = n EW StringBuffer (); for (int i = 0; i < num2; i++) {int index = start-num2 + i + 1;if (Index >-1) {Sb.append (cs[ind EX]);} else {num2--;}} if (Set.contains (sb.tostring ())) {Reverseover = Num2-1;return sb.tostring ();} Num2--;} Return string.valueof (Cs[start]);} public static void Main (string[] args) {string[] words = new string[] {"We", "We Five", "five people a group", "one group"};    splitstring ss = new splitstring (); /* Add words to Thesaurus */ss.addword (words); String st = "We are five groups"; System.out.println ("sentence to be participle:" + st);/* Use two ways to participle, below I specify the maximum word length is 4*/string[] ss2 = Ss.reversesplit (St, 4); string[] Ss1 = Ss.positivesplit (St, 4); System.out.println ("Forward maximum match word segmentation algorithm Word result:" + arrays.tostring (SS1)); System.out.println ("Inverse maximum match word segmentation algorithm Word result:" + arrays.tostring (SS2));}}

Run results

Maximum Matching segmentation algorithm