Machine Learning-from maximum likelihood estimation to EM Algorithm

Maximum Likelihood Estimation:

What we have been using in college learning is actually awesome!

What is the maximum likelihood estimation?
Q: Given a group of observed data, there is another model with a parameter to be determined. How can we estimate this unknown parameter?

Observed Data (x1, Y1)... (Xn, yn) the undetermined model parameter is θ, and the model is f (x; θ ). At this time, we can use the observation data to estimate this θ.This is the maximum likelihood function estimation.

For example:

Suppose we have a bag with white balls and black balls in it, but we don't know how many of them are. At this time, we need to estimate the probability of getting a ball to be white? How can we estimate it? You can fetch one hundred times from the bag by putting it back in a row to see whether it is a white ball or a black ball. Assume that the number of white balls in the first round is 70, and the number of black balls is 30. Set the probability that the extraction is a white ball to P. Then the total probability of the first extraction is p (x; P)

P (x; P) = P (x1, x2. ...... x100; θ) = P (x1;θ)*P (X2;θ)........P (x100;θ)

= P70 * (1-p) 30

At this time, we hope to maximize this probability.

Export:LOGP (X; P) = LogP70 * (1-p) 30 if the other derivative is 0, P = 0.7 can be obtained (Similarly, it can be used in a continuous variable. At this time, it is the product of the probability density function so easy)

Is it easy, right! That's easy! In fact, the maximum likelihood estimation is inGiven a set of data and a undetermined parameter model, how can we determine the unknown parameters of this model to maximize the probability of the known data produced by this Determined Parameter Model?. Of course, here I just introduced an estimation method with only one unknown parameter. The same is true for multiple unknown parameters, that is, the likelihood function is used to evaluate the maximum value. In fact, not all likelihood functions can be imported. When the likelihood function cannot be imported, we need to make the L (θ) The largestθ.

EM Algorithm:

I believe that the likelihood function has been achieved. Then let's look at the advanced.

A probability model sometimes contains both observation variables and hidden variables. If we only observe the variables, we can use the maximum likelihood method (or Bayesian) to estimate the unknown parameters. However, if there are hidden variables, it cannot be simply solved. This requires the EM algorithm.

You may not be very familiar with this problem or understand what implicit variables mean. Here is an example (an example of referencing a statistical learning method ):

Three coins are marked as A, B, and C respectively, and the positive probabilities are P, Q, and K. the rule is as follows: first throw coin a. If it is positive, select B. Otherwise, select C, then the selected coin (B or C), and observe the result. The positive side is 1 and the opposite side is 0. The results of the independent repeat experiments for 10 times are as follows. We don't know whether to be positive or positive when throwing a coin. We only know the final result and ask how to estimate the values of P, Q, and K?

If we know which coin we are throwing, we can use the maximum likelihood estimation to estimate these parameters, but we do not know. Because there is a reason for P, so it cannot be estimated, thisP is a hidden variable.

Log (shard) = Σ LOGP (X;Bytes) =Σ LOGP (x, P;Bytes),Attention is the required Q, K to be determined parameter, X is the observation data,Because of this P, we cannot solve Max.Σ LOGP (X;Bytes).

For example, we investigated the height of boys and girls. Height must follow the Gaussian distribution. In the past, we could sample male students to find out the Gaussian distribution parameters. Girls are also, but if we only know the height of a certain person, we cannot know whether he is a male or female (implicit variable ), at this time, the likelihood function cannot be used for estimation. You can use the EM method at this time.

 

There are two steps: E and M:

Step E:

First, assign a random value to a required parameter, and then obtain the posterior probability of another implicit parameter. This is the expected calculation process. First, we can assign the initial values P, Q, and K of the model parameters to obtain the results from each data to the model.

Step m

The posterior probability of the obtained implicit parameter is used to estimate the traditional likelihood function and correct the required parameter. Iteration until the two parameters are the same

 

In fact, it can be simply understood that we do not know the model parameters (such as Gaussian distribution) in unsupervised clustering ), at this time, we will randomly assign a value to the undetermined parameters (U and ó) of the model ). Then we can calculate the data that belongs to that category. Then we re-estimate U and ó with these classified data.

 

 

Reference: http://www.cnblogs.com/jerrylead/archive/2011/04/06/2006936.html

 

Machine Learning-from maximum likelihood estimation to EM Algorithm