Python implements the FMM Algorithm for Chinese word segmentation,
This example describes how to implement the FMM Algorithm for Chinese Word Segmentation in python. Share it with you for your reference. The specific analysis is as follows:
The simplest idea Of The FMM algorithm is to use the greedy algorithm to find n words forward. If these n words appear in the dictionary, OK. If they do not appear, find n-1... then proceed. If n words appear in the dictionary, continue searching from n + 1 until the sentence ends.
Import re def PreProcess (sentence, edcode = "UTF-8"): sentence = sentence. decode (edcode) sentence = re. sub (u "[.,,!......! "<> \"'::? \?, \ | "" '';]", "", Sentence) return sentence def FMM (sentence, diction, result = [], maxwordLength = 4, edcode = "UTF-8 "): I = 0 sentence = PreProcess (sentence, edcode) length = len (sentence) while I <length: # find the ascii word tempi = I tok = sentence [I: I + 1] while re. search ("[0-9A-Za-z \-\ + # @ _ \.] {1} ", tok) <> None: I = I + 1 tok = sentence [I: I + 1] if I-tempi> 0: result. append (sentence [tempi: I]. lower (). encode (edcode ))# Find chinese word left = len (sentence [I:]) if left = 1: "go to 4 step over the FMM" shocould we add the last one? Yes, if not blank "if sentence [I:] <>" ": result. append (sentence [I:]. encode (edcode) return result m = min (left, maxwordLength) for j in xrange (m, 0,-1): leftword = sentence [I: j + I]. encode (edcode) # print leftword. decode (edcode) if LookUp (leftword, diction): # find the left word in dictionary # it's the right one I = j + I result. append (leftword) break elif j = 1: "only one word, add into result, If not blank "if leftword. decode (edcode) <> "": result. append (leftword) I = I + 1 else: continue return result def LookUp (word, dictionary): if dictionary. has_key (word): return True return False def ConvertGBKtoUTF (sentence): return sentence. decode ('gbk '). encode ('utf-8 ') dictions = {} dictions ["AB"] = 1 dictions ["cd"] = 2 dictions ["abc"] = 1 dictions ["ss"] = 1 dictions [ConvertGBKtoUTF ("Okay")] = 1 di Ctions [ConvertGBKtoUTF ("true")] = 1 sentence = "asdfa. Is that good? vasdiw. Is it true that daf dasfiw asid? "S = FMM (ConvertGBKtoUTF (sentence), dictions) for I in s: print I. decode ("UTF-8") test = open ("test.txt", "r") for line in test: s = FMM (CovertGBKtoUTF (line), dictions) for I in s: print I. decode ("UTF-8 ")
The running result is as follows:
Asdfa
Okay.
Yes
This
Sample
?
Vasdiw
Ah
True
Daf
Dasfiw
Asid
Yes
?
?
I hope this article will help you with Python programming.