Maximum probability participle

Here is a method of word segmentation-the maximum probability participle, also called 1-gram participle, because it does not consider the context, only consider the current word probability.

We need to have a dictionary that records the frequency of each word, such as:

Based on this dictionary, we can express a sentence with a directed acyclic graph (DAG), such as

In this figure, each node is a word, and the edge is the probability of a two-point constituent word. The problem with the word breaker is to find out the probability and the largest one in the DAG that overwrites all the words from start to finish. For example, spicy-"dish, five-" meat, lid-"rice is a path, spicy-" spicy, white-"dish, five-" flower, meat-"meat, lid-" Rice is also a path, how to find the largest one?

Set \ (\alpha_i\) is the probability of the optimal path to the portion of the DAG that follows the I node, and J is the adjacency point of I, so easy to go out \ (\alpha_i = \SUM_JP (<i,j>) \alpha_j\),p<i,j> is the probability of a word.

This is a dynamic programming problem, starting with the last word, and gradually moving forward until the first point, and then getting the optimal path from the previous one.

The Python code is as follows:

defBuild_dag (sentence): DAG= {}#Dict,key is a word that each word's index,value is able to form with this word listN =Len (sentence) forKinchxrange (N): TMP=[] I=k piece=Sentence[k] whileI < N andPieceinchDict. FREQ:ifDict. Freq[piece]: tmp.append (i) I+ = 1piece= sentence[k:i + 1]       if  notTmp:tmp.append (k) dag[k]=tmpreturnDag

def Calc_route (sentence, DAG, route):         = Len (sentence)        = (0, 0)        = log (total_freq)        for in xrange ( N-1,-1,-1):            or 1)-                              for in Dag[idx])

def __cut_dag (self, sentence):     = Build_dag (sentence)    = {}    calc_route (DAG, route)    = 0    = Len ( Sentence)    = []    while x < N:        = route[x][1] + 1         = Sentence[x,y]        segs.append (word)        = y

Maximum probability participle