Python-based FMM Algorithm for Chinese Word Segmentation

This article mainly introduces how to implement the FMM Algorithm for Chinese Word Segmentation in python. The example shows how to implement the FMM Algorithm for Python based on Chinese word segmentation. It involves Python's skills for file, string, and regular expression matching operations, for more information about how to use python to implement The FMM algorithm, see the following example. Share it with you for your reference. The specific analysis is as follows:

The simplest idea Of The FMM algorithm is to use the greedy algorithm to find n words forward. If these n words appear in the dictionary, OK. If they do not appear, find n-1... then proceed. If n words appear in the dictionary, continue searching from n + 1 until the sentence ends.

Import re def PreProcess (sentence, edcode = "UTF-8"): sentence = sentence. decode (edcode) sentence = re. sub (u "[.,,!......! "<> \"'::? \?, \ | "" '';]", "", Sentence) return sentence def FMM (sentence, diction, result = [], maxwordLength = 4, edcode = "UTF-8 "): I = 0 sentence = PreProcess (sentence, edcode) length = len (sentence) while I <length: # find the ascii word tempi = I tok = sentence [I: I + 1] while re. search ("[0-9A-Za-z \-\ + # @ _ \.] {1} ", tok) <> None: I = I + 1 tok = sentence [I: I + 1] if I-tempi> 0: result. append (sentence [tempi: I]. lower (). encode (edcode ))# Find chinese word left = len (sentence [I:]) if left = 1: "go to 4 step over the FMM" shocould we add the last one? Yes, if not blank "if sentence [I:] <>" ": result. append (sentence [I:]. encode (edcode) return result m = min (left, maxwordLength) for j in xrange (m, 0,-1): leftword = sentence [I: j + I]. encode (edcode) # print leftword. decode (edcode) if LookUp (leftword, diction): # find the left word in dictionary # it's the right one I = j + I result. append (leftword) break elif j = 1: "only one word, add into result, If not blank "if leftword. decode (edcode) <> "": result. append (leftword) I = I + 1 else: continue return result def LookUp (word, dictionary): if dictionary. has_key (word): return True return False def ConvertGBKtoUTF (sentence): return sentence. decode ('gbk '). encode ('utf-8 ') dictions = {} dictions ["AB"] = 1 dictions ["cd"] = 2 dictions ["abc"] = 1 dictions ["ss"] = 1 dictions [ConvertGBKtoUTF ("Okay")] = 1 di Ctions [ConvertGBKtoUTF ("true")] = 1 sentence = "asdfa. Is that good? vasdiw. Is it true that daf dasfiw asid? "S = FMM (ConvertGBKtoUTF (sentence), dictions) for I in s: print I. decode ("UTF-8") test = open ("test.txt", "r") for line in test: s = FMM (CovertGBKtoUTF (line), dictions) for I in s: print I. decode ("UTF-8 ")

The running result is as follows:

Asdfa
Okay.
Yes
This
Sample
?
Vasdiw
Ah
True
Daf
Dasfiw
Asid
Yes
?
?

I hope this article will help you with Python programming.