"Turn" Gaussian mixture model

In this paper, how to establish this problem in Gaussian mixture model (Gmm,gaussian Mixture models), the implementation process of the expected maximization (Em,expectation maximization) algorithm is explained in detail.

Gaussian distribution Model GSM

When a multidimensional variable x obeys a Gaussian distribution, its probability density function PDF is:

x is the column vector of dimension D, U is the model expectation, Σ is the model variance. In practice, U is usually substituted with sample mean, Σ is usually substituted by sample variance. It is easy to tell if a sample of x belongs to category C. Because each class has its own u and σ, the X into the (1), when the probability is greater than a certain threshold, we think that x belongs to Class C.

Geometrically speaking, the Gaussian distribution model in two-dimensional space should approximate to the ellipse, in three-dimensional space approximation to the ellipsoid. Unfortunately, in many classification problems, sample points belonging to the same category do not meet the characteristics of the "ellipse" distribution. This introduces the Gaussian mixture model.

Gaussian mixture Model GMM

GMM thinks the data is generated from several GSM, namely

K needs to be determined beforehand, just like the K in K-means. Πk is a weight factor. Any one of these Gaussian distributions N (x;uk,σk) is called a component of the model. Here's the question, why should we assume that the data is made up of a number of Gaussian distributions, not other distributions? In fact, no matter what the distribution, only K is large enough, the XX Mixture model will become sufficiently complex, it can be used to approximate any continuous probability density distribution. Just because the Gaussian function has good computational performance, GMM is widely used.

GMM is a clustering algorithm, and each component is a clustering center. The model parameters (Π,u and σ) are computed in the case of only the sample points, without knowing the sample classification (which contains the implied variables)----which can obviously be solved by the EM algorithm. Then use the training model to distinguish the classification of the sample, the method is: Step1 randomly selected one of the K component (the probability of being selected is Πk), step2 the sample into the newly selected component, determine whether it belongs to this category, if not belong to the Step1.

Sample classification of GMM in known cases

When each sample belongs to the classification known, GMM's parameters are very well determined, directly using maximum likelihood. The sample size for the K classification is n1,n2,..., Nk, and the sample set belonging to the K-category is L (k).

GMM in case of unknown sample classification

There are n number of points, subject to some kind of distribution PR (x;θ), we want to find a set of parameters θ, so that the probability of generating these data point is the largest, the probability is

Called a likelihood function (Lilelihood functions). Usually the probability of a single point is very small, even after the data will be smaller, easy to cause floating point overflow, so generally take its logarithm, into

Called Log-likelihood function.

The Log-likelihood function of GMM is:

Here each sample Xi belongs to the category ZK is not aware of. Z is an implied variable.

We are going to find the best model parameters so that (6) the desired maximum, the "expected maximization algorithm" name from this.

Em method for solving

The general form of the problem of EM requirement solution is

Y is an implied variable.

We already know that if the classification label y of the data point is known, then the solution Model parameter can be used directly with the maximum likelihood. The basic idea of the EM algorithm is to initialize a set of parameters θ (0) randomly, according to the posteriori probability PR (y| x;θ) to update Y's expectation E (y) and then use E (y) instead of Y to find the new model parameter θ (1). So iterate until θ tends to stabilize.

E-step E is the meaning of expectation, that is, assuming that the model parameters are known, the implicit variable z takes z1,z2 respectively,... The expectation, that is, Z to take Z1,Z2 separately,... The probability. In GMM, the probability of the data points being generated by each component is obtained.

Notice that we multiply the posterior probability of z by a weight factor αk, which indicates the frequency of the data points belonging to the class ZK in the training set, and in GMM it is πk.

M-step m is the meaning of maximization, which is to use the maximum likelihood method to find out the model parameters. Now we think that the R (I,K) obtained in the previous step is the probability that data point Xi is generated by component K. According to the formula (3), (4), (5) can be introduced:

Comparison with K-means

The same point: both are available for clustering, and you need to specify a K value.

Different points: GMM can give a sample of what the probability of a class.

Reference: Http://www.cnblogs.com/zhangchaoyang

"Turn" Gaussian mixture model