"Reprint" Mixed Gaussian model (mixtures of Gaussians) and EM algorithm

Mixed Gaussian model (mixtures of Gaussians) and EM algorithm

This discussion uses the desired maximization algorithm (expectation-maximization) for density estimation (density estimation).

As with K-means, given the training sample, we will show the implied category label. Unlike the hard designation of K-means, we first think that it satisfies a certain probability distribution, and here we think that satisfies the polynomial distribution, where there are k values {1,..., k} can be selected. And we think that after a given, the multi-valued Gaussian distribution is satisfied. Thus the joint distribution can be obtained.

The whole model is simply described as for each example, we first extract a polynomial distribution from the K category, and then, based on one of the corresponding K multivalued Gaussian distributions, a sample is generated. The entire process is called a mixed Gaussian model. Note that here is still the implied random variable. The model also has three variables and. The maximum likelihood estimate is. Log in as follows:

The maximum value of this equation cannot be solved by using the 0 derivation method previously, because the result of the request is not the close form. But assuming that we know each example, the upper formula can be simplified to:

At this point, we'll come back and take the derivative:

is the ratio in the sample category. Is the sample characteristic mean of the class J, which is the covariance matrix of the characteristics of the class J sample.

In fact, when known, the maximum likelihood estimate is approximate to the Gaussian discriminant analysis model (Gaussian discriminant analytical models). The difference is that the Gda category Y is the Bernoulli distribution, where z is the polynomial distribution, and each sample here has a different covariance matrix, whereas Gda thinks there is only one.

Before we were given the hypothesis, it was actually not known. So what do we do? Considering the thought of EM as mentioned earlier, the first step is to guess the implied class variable Z, and the second step is to update the other parameters to get the maximum likelihood estimate. The use of this is:

Loop the following steps until convergence: { (e-Step) for each I and J, calculate (M step), update the parameters: }

In e-step, we consider the other parameters as constants, the computed posterior probabilities, that is, the estimation of implied class variables. After estimating, the above formula is used to recalculate the other parameters, and when the maximum likelihood estimation is found, the value is wrong and needs to be recalculated again and again until it converges.

The specific calculation formula is as follows:

This equation uses the Bayesian formula.

Here we use instead of the preceding, from a simple 0/1 value into a probability value.

Comparison K-means can be found here using the "soft" designation, for each sample assigned to a certain probability of the class, while the calculation is also larger, each sample I have to calculate the probability of each category J. As with K-means, the result is still the local optimal solution. It is a good idea to take different initial values of other parameters for multiple calculations.

Although the convergence of EM is described qualitatively in the previous K-means, it is still not given quantitatively, and the derivation process of generalized em is still not given. The next article focuses on these topics.

"Reprint" Mixed Gaussian model (mixtures of Gaussians) and EM algorithm