EM maximum expectation algorithm

References:http://blog.csdn.net/zouxy09/article/details/8537620
http://www.cnblogs.com/jerrylead/archive/2011/04/06/2006936.htmlMy Data Mining algorithm code implementation: Https://github.com/linyiqun/DataMiningAlgorithmIntroduction

The EM algorithm is an iterative algorithm for maximum likelihood estimation or maximal posteriori probability estimation of parametric models with implicit variables. EM algorithm, as a framework idea, it can be applied in many fields, such as data clustering----fuzzy clustering processing, will give an example of such implementation later.

Principle of EM algorithm

The EM algorithm can be seen from the name to be divided into 2 parts, E-step and M-step. E-step is called the desired step, and the m-step is the maximized step.

The steps for the overall algorithm are as follows:

1, initialize the distribution parameters.

2, (E-step) calculates the expected E, using the existing estimates of the hidden variables to calculate their maximum likelihood estimates, so as to achieve the desired process.

3, (M-step) maximize the maximum likelihood estimate on the e-step to calculate the value of the parameter

4, repeat 2,3 steps until convergence.

The above is the core principle of the EM algorithm, perhaps you will think, really so simple, in fact, I omitted the complex data derivation process, because if you do not understand the algorithm principle of EM, to see the derivation of the data formula, will make people more dizzy. OK, the following gives the data derivation process, I math is not good, so with the other people's derivation process, others have written very detailed.

the derivation process of EM algorithmJensen Inequalities

When introducing the derivation process, it is necessary to understand the Jensen inequality, which is a theorem about convex function, which is defined directly on the formula.

If f is a convex function and x is a random variable, then

In particular, if f is a strictly convex function, then if and only if, that is, X is a constant.

Here we will shorthand for.

If you use a diagram, it will be clear:

what needs to be explained here is that the value of E (X) is what is (a+b)/2, because 0.5 of the probability is a,0.5 is B, so his expectation is a A, a and the median. The same is true for values on the y-axis. formula expression form of EM algorithm

The EM algorithm translates to a formula in the form of:

Given the training sample is, the sample is independent, we want to find each sample implied category Z, can make P (x,z) the largest. The maximum likelihood estimates for P (X,Z) are as follows:

then make a little change to this formula, you can use the Jensen inequality, the magic of a pen to:

The following formula can be obtained from the previous description:

       (1) to (2) more directly, is the numerator denominator multiplied by an equal function. (2) to (3) using the Jensen inequality. The condition to be satisfied is

Again, since Q is the probability density function of the random variable z (i) , it can be obtained: the numerator sum equals c(the numerator denominator sums all Z (i) : the numerator denominator of multiple equations is added to the same, This thought that each sample two probability ratio is C), again continue to deduce;

Finally, the general process of EM algorithm is obtained:

Cycle repeats until convergence

(e step) for each I, calculate

(M-Step) calculation

Maybe you read this mathematical deduction process has begun to dizzy, no relationship, the following gives an example, so that we really feel the magic of the EM algorithm. Fuzzy clustering implementation of EM algorithm

Here I will give a self-realized fuzzy clustering based on EM algorithm.

Enter the test data file, which contains the A-f 7 point coordinates:

3 34 109 614 818 1121 7

at the beginning, the default cluster center point C1, C2 A and B. This is the initial assignment of parameters, followed by the main operation;

1. E-step: The desired step is to assign the object to the cluster according to the current fuzzy clustering or the parameters of the probability cluster.

2, M-step: Maximize the step to discover new clustering or parameters, minimize the fuzzy clustering of SSE (the object's error squared sum, which will be reflected in the program). This formula is used in M-step to readjust the center of the computed cluster according to the dividing Matrix.

The final convergence condition is that the calculated error and no more than 1.0 of the coordinate axis of the center point of the cluster, meaning that the basic no longer changes.
Main Program class:

Package Datamining_em;import Java.io.bufferedreader;import Java.io.file;import java.io.filereader;import Java.io.ioexception;import java.text.messageformat;import java.util.arraylist;/** * EM max expectation algorithm Tool class * * @author Lyq * */PUBL IC class Emtool {//test data file address private String datafilepath;//test coordinate point data private string[][] data;//test coordinate point data list private ARRAYLIST&L T Point> pointarray;//target C1 points private point p1;//destination C2 points Private point p2;public Emtool (String datafilepath) { This.datafilepath = Datafilepath;pointarray = new arraylist<> ();} /** * Read data from file */public void Readdatafile () {File File = new file (DataFilePath); arraylist<string[]> DataArray = new arraylist<string[]> (); try {bufferedreader in = new BufferedReader (new FileReader (file)); String str; String[] Temparray;while ((str = in.readline ()) = null) {Temparray = Str.split ("");d Ataarray.add (Temparray);} In.close ();} catch (IOException e) {e.getstacktrace ();} data = new String[dataarray.size ()] [];d ataarray.toarray (data);//start with a default of 2 points as 2Cluster center P1 = new Point (Integer.parseint (data[0][0]), Integer.parseint (Data[0][1]));p 2 = new Point (Integer.parseint (data[1) [0]), Integer.parseint (data[1][1]); Point P;for (string[] array:data) {//Convert data to object join list for convenient calculation P = new Point (Integer.parseint (array[0]), Integer.parseint (array[ 1]));p Ointarray.add (P);}} /** * Calculates the degree of membership of a coordinate point for a 2-cluster center point * * @param p * The coordinates to be tested */private void computemembership (point P) {//P points distance from the center point of the first cluster Doub Le Distance1 = 0;//p distance from the second center point double distance2 = 0;//is calculated with European distance Distance1 = Math.pow (P.getx ()-P1.getx (), 2) + Math.pow (P.G Ety ()-P1.gety (), 2);d Istance2 = Math.pow (P.getx ()-P2.getx (), 2) + Math.pow (p.gety ()-P2.gety (), 2);//Calculate the degree of membership for P1 points, and distance Inverse relationship, the smaller the distance, the greater the degree of membership, so to use a large distance2 additional distance to represent P.SETMEMBERSHIP1 (Distance2/(Distance1 + distance2));// Calculate the membership degree for P2 points p.setmembership2 (Distance1/(Distance1 + Distance2));} /** * Perform desired maximization step */public void Exceptmaxstep () {//New optimized cluster center point double p1x = 0;double p1y = 0;double P2X = 0;double p2y = 0;do Uble Temp1 = 0;double Temp2 = 0;//error value double Errorvalue1 = 0;double ErrorValue2 = 0;//The last updated cluster point coordinate, dot lastP1 = null; Point lastP2 = null;//When the calculation is started, or if the error value of the center point exceeds 1, you need to iterate the computation again while (lastP1 = = NULL | | errorValue1 > 1.0 | | errorValue2 > 1.0) {for (point P:pointarray) {computemembership (P);p 1X + = P.GETMEMBERSHIP1 () * P.GETMEMBERSHIP1 () * P.GETX ();p 1Y + = p. GETMEMBERSHIP1 () * P.GETMEMBERSHIP1 () * p.gety () Temp1 + = P.getmembership1 () * P.GETMEMBERSHIP1 ();p 2X + = P.GETMEMBERSHIP2 () * P.GETMEMBERSHIP2 () * P.GETX ();p 2Y + + p.getmembership2 () * P.GETMEMBERSHIP2 () * p.gety (); Temp2 + = P.ge TMEMBERSHIP2 () * P.GETMEMBERSHIP2 ();} LastP1 = new points (P1.getx (), P1.gety ()), lastP2 = new Point (P2.getx (), P2.gety ()), and/or sets of formulas for calculating new cluster Center coordinates, maximizing processing p1.setx (p1x/te MP1);p 1.setY (P1Y/TEMP1);p 2.setX (P2X/TEMP2);p 2.setY (P2Y/TEMP2), errorValue1 = Math.Abs (Lastp1.getx ()-P1.getx ()) + Ma Th.abs (Lastp1.gety ()-p1.gety ()) errorValue2 = Math.Abs (Lastp2.getx ()-P2.getx ()) + Math.Abs (lastp2.gety ()-p2.gety () );} System.out.println (Messageformat.format ("Cluster Center node P1 ({0}, {1}), p2 ({2}, {3}) ", P1.getx (), P1.gety (), P2.getx (), P2.gety ()));} 

Coordinate Point class:

/** * Coordinate points class *  * @author lyq *  */public class Point {//coordinates points horizontal axis private double x;//coordinate point ordinate private double y;//coordinate point for P1  Membership private Double membership1;//coordinate points for P2 the membership of private double Membership2;public point (double D, double e) {this.x = D;this.y = e;} Public double GetX () {return x;} public void SetX (double x) {this.x = x;} Public double GetY () {return y;} public void sety (double y) {this.y = y;} Public double GetMemberShip1 () {return memberShip1;} public void SetMemberShip1 (double memberShip1) {this.membership1 = MemberShip1;} Public double getMemberShip2 () {return memberShip2;} public void SetMemberShip2 (double memberShip2) {this.membership2 = memberShip2;}}

Call class;

/** * EM expectation maximization algorithm scenario call class * @author Lyq * */public class Client {public static void main (string[] args) {String FilePath = "c:\\ Users\\lyq\\desktop\\icon\\input.txt "; Emtool tool = new Emtool (FilePath); Tool.readdatafile (); Tool.exceptmaxstep ();}}

Output Result:

Cluster Center node P1 (7.608, 5.907), p2 (14.208, 8.745)

In this program, the hidden variable is the cluster center point, through constant iterative calculation, the final infinite close to the real value, quite interesting algorithm.

EM maximum expectation algorithm