MapReduce Implements linear regression

1. Software version:

Hadoop2.6.0 (idea in source code compiled using CDH5.7.3, corresponding to Hadoop2.6.0), the cluster uses the native Hadoop2.6.4,jdk1.8,intellij idea 14. Source code can be downloaded in https://github.com/fansy1990/linear_regression.

2. Realize the idea: This blog realizes is a one-dimensional linear equation, is equal to the simplest linear equation, the use of Couresa inside the machine learning Big Data linear equation method to update the parameter values (that is, the random gradient descent method, of course, can also be achieved using the batch gradient descent method, It's not the same as it does in linearregressionjob), and if you don't understand the random gradient drop or the batch gradient drop, you need to take a look first. Here's the idea: 2.1 Shuffle data: If you want to use a random gradient drop, you need to keep the original data random, so the first step here is to randomly disrupt the original data. The idea is to output the random value as key at the mapper end, output the current record as value, traverse all values of each key directly on the reducer side, and directly output value and Nullwritable.get. Add an additional parameter Randn here, which indicates how many raw data uses the same random value at the mapper end, and if the RANDN is 1, then each raw data will use a random value as key, and if RANDN is 2, So every two raw data using a random value, if the RANDN is 0 or less than 0, then all the data use the same random value (note, this time in fact, at the reducer end of the values are actually disorderly, please readers think why?) ）。 The map core implementations in its mapper are shown below

protected void Map (longwritable key, Text value, Context context) throws IOException, interruptedexception {        if (randn <= 0) {//If RANDN is smaller than 0, then do not disturb data            context.write again (randfloatkey,value);            return;        }        if (++counti >= randn) {//if RANDN equals 1, then each random value is not the same            Randfloatkey.set (Random.nextfloat ());            Counti =0;        }        Context.write (Randfloatkey,value);    }

2.2 Linear Regression (linear regression): Linear regression uses a random gradient descent method to update the THETA0 and theta1 (only one unary once, so only two parameters), and each mapper uses the same initialization parameters (theta0= 1 and theta1=0), use your own data in each mapper to update theta0 and THETA1, and the updated formula is:

THETA0 = theta0-alpha* (h (x)-y) xtheta1 = theta1-alpha* (h (x)-y) x

where h (x) = theta0 + theta1 * x; At the same time, be aware that the update here is a synchronous update with the core code shown below:

protected void Map (longwritable key, Text value, Context context) throws IOException, interruptedexception {        float[] X y = utils.str2float (Value.tostring (). Split (splitter));        float x = xy[0];        Float y = xy[1];        Synchronous update theta0 and theta1        lastTheta0 = theta0;        THETA0  -= Alpha * (theta0+theta1* x-y) * ×;//keep theta0 and theta1 unchanged        theta1-= Alpha * (lastTheta0 + theta1 * x-y) * x;//keep theta0 and theta1 unchanged    }

Then output the theta parameter values directly in the cleanup function of each mapper.

protected void Cleanup (context context) throws IOException, interruptedexception {        theta0_1.set (theta0 + splitter + t HETA1);        Context.write (Theta0_1,nullwritable.get ());    }

Since each parameter value of theta has been updated in each mapper, it is not necessary to use reducer, and because the test data is relatively small, So to set the size of the mapreduce.input.fileinputformat.split.maxsize, the reader needs to be set according to the size of its actual data, and its driver class core code is as follows:

Conf.setlong ("Mapreduce.input.fileinputformat.split.maxsize", 700L);//Get multiple mapper;job.setnumreducetasks (0);

2.3 Combine Theta (merge parameter values): In 2.2 Steps have been calculated for the various Theta values, then how to merge these obtained Theta values? Can you use the average value directly? For unary one-time linear regression, the average value can be used directly as the final merged theta, but for other linear regression (specifically, linear regression with multiple local minimums, there is a problem with multiple theta values being merged). If you only use the average, then in 2.2 steps to add a reducer can be completed, here is an alternative way to merge the theta value, that is, using the global error of each theta value as a parameter to be weighted. Therefore, in the mapper setup will read more than 2.2 output theta value, in the map function for each raw data to find its error, the output to reducer data is theta value and its error, the core code is as follows:

protected void Map (longwritable key, Text value, Context context) throws IOException, interruptedexception {        float[] X y = utils.str2float (Value.tostring (). Split (splitter));        for (int i =0;i<thetas.size (); i++) {            //error = (theta0 + theta1 * x-y) ^2            thetaerrors[i] + = (Thetas.get (i) [0] + Thetas.get (i) [1] * xy[0]-xy[1]) *                    (Thetas.get (i) [0]+ thetas.get (i) [1] * xy[0]-xy[1]);            thetanumbers[i]+= 1;        }    }

protected void Cleanup (context context) throws IOException, interruptedexception {for        (int i =0;i<thetas.size (); i++) {            theta.set (Thetas.get (i));            Floatandlong.set (Thetaerrors[i],thetanumbers[i]);            Context.write (Theta,floatandlong);        }    }

On the reducer side, each error is added directly to each key (that is, the theta value), and the Theta value is combined in the cleanup function with weights, and its core code is as follows:

protected void reduce (floatandfloat key, iterable<floatandlong> values, context context) throws IOException, interruptedexception {        float sumf = 0.0f;        Long SumL = 0L;        for (Floatandlong value:values) {            sumf +=value.getsumfloat ();            SumL + = Value.getsumlong ();        }        Theta_error.add (New Float[]{key.gettheta0 (), Key.gettheta1 (), (float) math.sqrt ((double) sumf/suml)});        Logger.info ("theta:{}, error:{}", New Object[]{key.tostring (), Math.sqrt (SUMF/SUML)});    

protected void Cleanup (context context) throws IOException, interruptedexception {//how weighted?        Mode 1: If the error is smaller, then the weight should be greater;//Mode 2: Direct mean float [] Theta_all = new float[2];            if ("average". Equals (method)) {//Theta_all = Theta_error.get (0);                for (int i=0;i< theta_error.size (); i++) {theta_all[0] + = Theta_error.get (i) [0];            THETA_ALL[1] + = Theta_error.get (i) [1];            } Theta_all[0]/= theta_error.size ();        THETA_ALL[1]/= theta_error.size ();            } else {float sumerrors = 0.0f;            For (float[] d:theta_error) {sumerrors + = 1/d[2];                } for (float[] d:theta_error) {theta_all[0] + = d[0] * 1/d[2]/sumerrors;            THETA_ALL[1] + = d[1] * 1/d[2]/sumerrors;    }} context.write (new Floatandfloat (Theta_all), Nullwritable.get ()); }

2.4 Verify that the validation here refers to the use of 2.3 steps to obtain the combined theta value of the global error, because in 2.3 steps also obtained the global error of each theta value, so here can compare to see which theta value is optimal, its mapper can directly use 2.3 step mapper, and reducer is similar to 2.3 The reducer in the step, but the final output does not require merging in the cleanup. 3. Running Result: 3.1 Shuffle Job test class:

public static void Main (string[] args) throws Exception {        args = new string[]{                "hdfs://master:8020/user/fanzhe/ Linear_regression.txt ",                " Hdfs://master:8020/user/fanzhe/shuffle_out ",                " 1 "        }    ;        Toolrunner.run (utils.getconf (), New Shuffledatajob (), args);    

Raw data: (Can download linear_regression.txt in the source resource directory)

6.1101,17.5925.5277,9.13028.5186,13.662 ...

Shuffle output: Each time the output should be different (using a random number), you can see that the data is indeed randomized. 3.2 Linear regression test class:

public static void Main (string[] args) throws Exception {//        <input> <output> <theta0;theta1;alpha > <splitter>//Note The third parameter uses a semicolon        to split the args = new string[]{                "Hdfs://master:8020/user/fanzhe/shuffle_out",                "Hdfs://master:8020/user/fanzhe/linear_regression",                "1;0;0.01",                ","        }    ;        Toolrunner.run (utils.getconf (), New Linearregressionjob (), args);    

View the output results: From the output can be seen, two results difference is still very large, this is mainly because the test data relatively small reason, if the data is relatively large, and is very good shuffle, then the two values should be small difference; 3.3 Combine theta Test class:

public static void Main (string[] args) throws Exception {//        <input> <output> <theta_path> < Splitter> <average|weight>        args = new string[]{                "Hdfs://master:8020/user/fanzhe/shuffle_out", "                hdfs://master:8020/user/fanzhe/single_linear_regression_error",                "hdfs://master:8020/user/fanzhe/ Linear_regression ",                ", ","                Weight "        }    ;        Toolrunner.run (utils.getconf (), New Singlelinearregressionerror (), args);    

The method of merging Theta values here is weighted, and the reader can be set to average, thus using the mean; Results:
According to the log can be seen theta parameter values to select one of the following, its error will be relatively small, the combined parameter value is:
See the result is between the two theta parameter values. If it is an average, then its output is:
3.4 Validation Validation test class:

public static void Main (string[] args) throws Exception {//        <input> <output> <theta_path> < splitter>        args = new string[]{                "Hdfs://master:8020/user/fanzhe/shuffle_out",                "hdfs://master:8020/ User/fanzhe/last_linear_regression_error ",                " Hdfs://master:8020/user/fanzhe/single_linear_regression_error " ,                ",",        }    ;        Toolrunner.run (utils.getconf (), New Lastlinearregressionerror (), args);    

The output is:
It can be seen from the results that the combined results do not have the same effect as one of the theta parameter group values, but this may also be related to the data volume, depending on the output, you can also combine the theta value and the pre-merger comparison, and then use the optimal theta as the final output. If it is an average, then its output is:

From the above results, we can see that the weighted combination is better than the average combination effect; 4. Summary 1. The algorithm is only for the case that there is a local optimal solution (that is, the global optimal solution), otherwise, there will be a problem in the merging phase; 2. Through the small amount of data validation, the use of the combined effect does not use the optimal solution before merging the effect is good, this may be a data problem, to be verified; 3. Through the very intuitive imagination, the use of weighted combination in general is better than the average group good effect;

Share, grow, be happy

Reprint Please specify blog address: http://blog.csdn.net/fansy1990

MapReduce Implements linear regression