Hmm memm & label bias

Http://blog.csdn.net/xum2008/article/details/38147425)

Hidden Markov Model (HMM ):

Figure 1 Hidden Markov Model

Disadvantages of Hidden Markov Model:

1. Hmm only depends on each State and its corresponding observed object:

Sequence labeling is not only related to a single word, but also to the length of the observed sequence, the context of the word, and so on.

2. the target function does not match the prediction target function:

Hmm learns the joint distribution of State and observation sequence P (Y, x). In the prediction problem, what we need is the conditional probability P (Y | X ).

 

Maximum Entropy Hidden Markov Model (memm ):

 

 

Figure 2. Maximum Entropy Markov Model

Memm takes into account the dependency between adjacent states and the entire observation sequence, so memm has a stronger expression capability. memm does not consider p (x) to reduce the modeling burden, at the same time, we learned that the target function is consistent with the prediction function.

Memm tag Bias:

 

 

Figure 3. the Viterbi algorithm decodes memm. State 1 tends to be converted to State 2, and state 2 tends to be retained in state 2;

P (1-> 1-> 1-> 1) = 0.4x0.45x0.5 = 0.09, P (2-> 2-> 2-> 2) = 0.2x0.3x0.3 = 0.018,

P (1-> 2-> 1-> 2) = 0.6x0.2x0.5 = 0.06, P (1-> 1-> 2-> 2) = 0.4x0.55X0.3 = 0.066.

In Figure 3, state 1 tends to be converted to State 2, and state 2 tends to be retained in state 2; but the optimal state conversion path is 1-> 1-> 1, why? Because status 2 can be converted to more States than status 1, so that the transfer probability is reduced; that is, memm tends to select a state with less transfer. This is the labeling bias problem.CRF effectively solves the labeling bias problem.

Memm is local normalization, while CRF is global normalization.

On the other hand, memms cannot find the corresponding parameter to meet the following distribution:

A B c --> A/a B/B c/c p (a B c | a B C) = 1

A B e --> A/A B/d E/e p (a d e | A B e) = 1

P (A | A) P (B | B, A) P (c | C, B) = 1

P (A | A) P (d | B, A) P (E | E, D) = 1

However, CRFs can find that the model meets this distribution.