Hamam Study Notes (2)

To write a hamajob, if you only write a hamajob, you don't need eclipse at all. All the code can be done in a java file. However, those who are familiar with eclipse are not familiar with vim and others. Create a userlibrary in eclipse: Window-Preferences-Java-BuildPath-UserLibraries In the eclipse menu bar

If you only write the hamn job, you do not need eclipse at all. All the code can be done in a java file. However, those who are familiar with eclipse are not familiar with vim and others. In eclipse, you can create a user library: In the eclipse menu bar: Window-Preferences-Java-Build Path-User Libraries

Write a Hamma job

If you only write the Hamma job, you do not need eclipse at all. All the code can be done in a java file. However, those who are familiar with eclipse are not familiar with vim and others.

You can create a user library in eclipse:

On the eclipse menu bar: Window-> Preferences-> Java-> Build Path-> User Libraries-> New create a user library, such as a hama-0.6.0, check System Library. Add External JARs to Add the jar package in HAMA_HOME/lib and the jar package in HAMA_HOME.

When you create a Java Project, add the user library to the Project. You can use the example of PI calculation in hamexample:

Http://blog.csdn.net/bhq2010/article/details/8513052]

import java.io.IOException;import org.apache.commons.logging.Log;import org.apache.commons.logging.LogFactory;import org.apache.hadoop.fs.FSDataInputStream;import org.apache.hadoop.fs.FileStatus;import org.apache.hadoop.fs.FileSystem;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.DoubleWritable;import org.apache.hadoop.io.IOUtils;import org.apache.hadoop.io.NullWritable;import org.apache.hadoop.io.Text;import org.apache.hama.HamaConfiguration;import org.apache.hama.bsp.BSP;import org.apache.hama.bsp.BSPJob;import org.apache.hama.bsp.BSPJobClient;import org.apache.hama.bsp.BSPPeer;import org.apache.hama.bsp.ClusterStatus;import org.apache.hama.bsp.FileOutputFormat;import org.apache.hama.bsp.NullInputFormat;import org.apache.hama.bsp.TextOutputFormat;import org.apache.hama.bsp.sync.SyncException;public class PiEstimator{    private static Path TMP_OUTPUT = new Path("/tmp/pi-"    + System.currentTimeMillis());    public static class MyEstimator    extends    BSP
 
      {public static final Log LOG = LogFactory.getLog(MyEstimator.class);private String masterTask;private static final int iterations = 10000;@Overridepublic void bsp(BSPPeer
  
    peer)throws IOException, SyncException, InterruptedException{    int in = 0;    for (int i = 0; i < iterations; i++)    {double x = 2.0 * Math.random() - 1.0, y = 2.0 * Math.random() - 1.0;if ((Math.sqrt(x * x + y * y) < 1.0)){    in++;}    }    double data = 4.0 * in / iterations;    peer.send(masterTask, new DoubleWritable(data));    peer.sync();}@Overridepublic void setup(BSPPeer
   
     peer)throws IOException{    // Choose one as a master    this.masterTask = peer.getPeerName(peer.getNumPeers() / 2);}@Overridepublic void cleanup(BSPPeer
    
      peer)throws IOException{    if (peer.getPeerName().equals(masterTask))    {double pi = 0.0;int numPeers = peer.getNumCurrentMessages();DoubleWritable received;while ((received = peer.getCurrentMessage()) != null){    pi += received.get();}pi = pi / numPeers;peer.write(new Text("Estimated value of PI is"),new DoubleWritable(pi));    }}    }    static void printOutput(HamaConfiguration conf) throws IOException    {FileSystem fs = FileSystem.get(conf);FileStatus[] files = fs.listStatus(TMP_OUTPUT);for (int i = 0; i < files.length; i++){    if (files[i].getLen() > 0)    {FSDataInputStream in = fs.open(files[i].getPath());IOUtils.copyBytes(in, System.out, conf, false);in.close();break;    }}fs.delete(TMP_OUTPUT, true);    }    public static void main(String[] args) throws InterruptedException,    IOException, ClassNotFoundException    {// BSP job configurationHamaConfiguration conf = new HamaConfiguration();BSPJob bsp = new BSPJob(conf, PiEstimator.class);// Set the job namebsp.setJobName("Pi Estimation Example");bsp.setBspClass(MyEstimator.class);bsp.setInputFormat(NullInputFormat.class);bsp.setOutputKeyClass(Text.class);bsp.setOutputValueClass(DoubleWritable.class);bsp.setOutputFormat(TextOutputFormat.class);FileOutputFormat.setOutputPath(bsp, TMP_OUTPUT);BSPJobClient jobClient = new BSPJobClient(conf);ClusterStatus cluster = jobClient.getClusterStatus(true);if (args.length > 0){    bsp.setNumBspTask(Integer.parseInt(args[0]));} else{    // Set to maximum    bsp.setNumBspTask(cluster.getMaxTasks());}long startTime = System.currentTimeMillis();if (bsp.waitForCompletion(true)){    printOutput(conf);    System.out.println("Job Finished in "    + (System.currentTimeMillis() - startTime) / 1000.0    + " seconds");}    }}
    
   
  
 

Run as Java Application, which runs in standalone mode and does not need to be installed or started. If you want to run the project on the cluster, you can convert the project Export into a Jar file and send it to the cluster for running.

Compile the hama source code

The source code project of Hamam is built with maven. Download and decompress the src package of Hamam;

Install m2e in eclipse to Import-> Maven-> Existing Maven Project-> select the directory where the decompressed Hamam source code is located to Import the file. During the first Import, maven downloads dependent packages, so it takes a long time.

After the import, there will be six projects, such as hama-core \ hama-graph. Then, you can use the maven plug-in to compile and debug the source code of the source code.

If you do not need eclipse, You need to download and install maven2, which can be found everywhere.

To save trouble, you can download the eclipse for jave EE developer of Juno version. This version of eclipse contains the complete m2e plug-in.