Data mining ten algorithms----EM algorithm (maximum expectation algorithm) __ algorithm

Concept

In statistical calculations, the maximal expectation (EM) algorithm is the algorithm for finding the maximum likelihood or maximum posteriori estimation of a parameter in the probability (probabilistic) model, where the probabilistic model relies on the invisible hidden variable (latent Variable).

Maximum expectations are often used in the field of data clustering for machine learning and computer vision.

can have some comparative image of the metaphor to make this algorithm clear.

For example, the canteen chef fried a dish, to be divided into two parts to two people to eat, it is clear that there is no need to take the balance 1.1 points of the accurate to weigh the components, the simplest way is to take the dishes into two bowls, and then observe whether it is the same as many, the more that one to take out a point in another bowl This process has been carried out iteratively until you can see no difference in the weight of the dishes that are contained in the two bowls. (from Baidu Encyclopedia)

The EM algorithm is like this, assuming we know that A and B two parameters, in the beginning of both are unknown, and know the information of a can get B information, in turn know B also got a. Consider first giving a certain initial value, in order to get the estimate of B, and then starting from the current value of B, re-estimate the value of a, the process continues until convergence.

The EM algorithm is also the basis for many unsupervised clustering algorithms (such as Cheeseman et al. 1988), and it is used to learn some observable Markov models (partially Observable Markov Model) is widely used on the basis of the Baum-welch front-backward algorithm. estimating mean values of K-Gaussian distributions

The most convenient way to introduce an EM algorithm is through an example.

Consider that data d is an instance collection, which is generated by a mixed-derived distribution of k different normal distributions. The problem framework is shown in the following figure, where k=2 and instances are points displayed along the x-axis.

Each instance is formed using a two step process.

First, randomly select one of the K normal distributions.

Then the random variable XI is generated according to the distribution of this selection.

This process repeats itself, generating a set of data points as shown in the figure. To simplify the discussion, we consider a simple case in which the selection of a single normal distribution is based on a uniform probability, and the K normal distribution has the same variance σ2, and σ2 is known.

The learning task is to output a hypothetical h=<μ1 ... Μk>, which describes the mean value of each distribution in K distributions. We want to find a maximum likelihood hypothesis for these mean values, a hypothetical H that maximizes P (d|h).

Note that when given a data instance extracted from a normal distribution x1,x2, ..., XM, it is easy to calculate the maximum likelihood assumption for the mean of the distribution.

Where we can prove that the maximum likelihood hypothesis is the assumption that the squared sum of errors on the M training instance is minimized.

Use when expressing a formula, you can get:

(Formula I)

However, our problem here involves a mix of k different normal distributions, and we don't know which one is the distribution. So this is a typical example of a hidden variable.

em algorithm steps

In the example above, the complete description of each instance can be thought of as ternary <xi,zi1, ZI2>, where Xi is the observed value of the I-instance, ZI1 and Zi2 indicate which of the two normal distributions is used to produce the value XI.

To be exact, Zij in Xi is produced at 1 by the first normal distribution, otherwise 0. Here Xi is the variable observed in the description of the instance, Zi1 and Zi2 are hidden variables. If the values of Zi1 and Zi2 are known, they can be used to solve the mean μ1 and μ2. Because they are unknown, we can only use the EM algorithm.

The EM algorithm is applied to our K-means problem, and the aim is to search for a maximum likelihood hypothesis, which is based on the current hypothesis <μ1 ... Μk> constantly re-estimate the expected value of the hidden variable zij. Then the maximum likelihood hypothesis is recalculated with the expected value of these hidden variables. This first describes the instantiation of the EM algorithm, the general form of the EM algorithm will be given later.

To estimate the two mean values in the above image, the EM algorithm first initializes the hypothesis to H=<μ1,μ2>, where μ1 and μ2 are arbitrary initial values. Then repeat the following two steps to re-estimate h until the process converges to a stable h value.

Step 1: Calculate the expected value of each hidden variable zij e[zij], assuming that the current hypothesis h=<μ1,μ2> established.

Step 2: Calculate a new maximum likelihood hypothesis h´=<μ1´,μ2´>, assuming that the value taken by each hidden variable zij is the expected e[zij in step 1th], and then replace the hypothesis h=<μ1,μ2> with the new hypothesis H´=<μ1´,μ2´ And then loop.

Now examine how the first step is achieved. Step 1 To calculate the expected value for each zij. This e[zij] is the probability that instance Xi is generated by the J Normal Distribution:

Therefore, the first step can be implemented by substituting the current value <μ1,μ2> and the known XI generation into the upper style.

In the second step, use the E[zij obtained in the 1th step] to derive a new maximum likelihood hypothesis h´=<μ1´,μ2´>. As will be discussed later, the maximum likelihood assumption is:

Note This expression is similar to the sample mean in Formula One, which is used to estimate μ from a single normal distribution. The new expression is just a weighted sample mean for the μj, with each instance weighted to its expected value from the J normal distribution.

The algorithm above estimates the K normal distribution mean, which describes the main points of the EM method : The current assumptions are used to estimate unknown variables, and the expected values of these variables are used to improve the assumptions.

It can be proved that in the first cycle of this algorithm, the EM algorithm can increase the likelihood P (d|h) unless it has reached the local maximum. Therefore, the algorithm converges to a local maximum probability hypothesis for <μ1,μ2>.

General description of EM algorithm

The EM algorithm above is for estimating the average value of the mixed normal distribution. More generally, the EM algorithm can be used in a number of problem frameworks, where a set of parameter θ that describes the baseline probability distribution needs to be estimated, given only the observable portion of all the data generated by this distribution.

In the above two mean problem, the parameter of interest is Θ=<Μ1,Μ2>, while all data is ternary <xi,zi1, ZI2>, and only Xi can be observed, generally x=<x1, ...,xm> represents the data that has been observed in the same instance and make Y=X∪Z representative of all data. Notice that the observed z can be considered as a random variable, and its probability distribution depends on the unknown parameter θ and the known data x. Similarly, y is a random variable, because it is defined by the random variable Z. In the subsequent sections, the general form of the EM algorithm is described. The h is used to represent the hypothetical value of the parameter θ, while H´ represents the assumptions modified in each iteration of the EM algorithm.

The EM algorithm seeks the maximum likelihood hypothesis H´ by searching for the largest h´ of E[LNP (Y|h´). This expectation is calculated on the probability distribution followed by Y, which is determined by the unknown parameter θ. Consider what this expression means.

First P (Y|h´) is the likelihood of all data Y under a given hypothesis H´. The rationality is that we are looking for a h´ to maximize the value of a function of this quantity.

Secondly, maximizing the logarithmic LNP (Y|h´) of this amount also maximizes P (Y|h´), as already described.

Thirdly, the introduction of expected value E[LNP (Y|h´)] is because all data Y itself is also a random variable.

It is known that all data y is a combination of observed X and non-observed z, and we must take an average of the possible values of the observed z, with the corresponding probability as the weighted value. In other words, take the expected value E[LNP (Y|h´)] on the probability distribution that the random variable Y follows. The distribution is determined by the fully known x value plus the distribution of Z obedience.

What is the probability distribution of y compliance. This distribution is generally not known because it is determined by the θ parameter to be estimated. However, the EM algorithm uses its current hypothesis h instead of the actual parameter θ to estimate the distribution of Y. Now defines a function q (h´|h), which gives E[LNP (Y|h´)] as a function of H´, under the assumption of the observed part X of θ=h and all data Y.

The Q function is written as Q (h´|h) to indicate that its definition is under the assumption that the current hypothesis H equals θ. In the general form of the EM algorithm, it repeats the following two steps until convergence.

Step 1: Estimate (E) Step: Use the current hypothesis H and the observed data x to estimate the probability distribution on y to calculate Q (h´|h).

Step 2: Maximize (M) Step: Replace the assumption H with the assumption that the Q function is maximized H´:

When the function q is continuous, the EM algorithm converges to a fixed point of the likelihood function P (y|h´). If the likelihood function has a single maximum value, the EM algorithm can converge to this global maximum likelihood estimate for H´. Otherwise, it only guarantees convergence to a local maximum value. Thus, EM has the same limitations as other optimization methods, such as gradient descent, linear search, and deformation gradients discussed earlier. In conclusion, the EM algorithm maximizes the logarithmic likelihood function of incomplete data by iteratively maximizing the expectation of the logarithmic likelihood function of the complete data.