Hadoop Eclipse Development Environment Building

First, install Eclipse

download Eclipse, unzip the installation, such as installing to /usr/local, namely/usr/local/eclipse

4.3.1 version: Http://pan.baidu.com/s/1eQkpRgu

Ii. Installing the Hadoop plugin on Eclipse

1. Download the Hadoop plugin

:Http://pan.baidu.com/s/1mgiHFok

This zip file contains the source code, we use the compiled jar can be extracted, the release folder in the Hadoop.eclipse-kepler-plugin-2.2.0.jar is a compiled plug-in.

2, put the plug-in into the Eclipse/plugins directory

3. Restart Eclipse, configure Hadoop installation directory

If the plug-in installation succeeds, after opening windows-preferences, there will be a Hadoop map/reduce option on the left side of the window, click this option to set the Hadoop installation path on the right side of the window.

4, Configuration Map/reduce Locations

Open Windows-open Perspective-other

Select Map/reduce, click OK

in the lower right, see as shown

Click on the Map/reduce Location tab and click on the icon on the right to open the Hadoop location Configuration window:

Enter location name, any name. Configure Map/reduce Master and DFS Mastrer,host and port to be configured to match core-site.xml settings.

Click the "Finish" button to close the window.

Click on the left Dfslocations->myhadoop (location name in the previous step), if you can see user, the installation is successful.

If the installation fails as shown, check to see if Hadoop is started and the eclipse is configured correctly.

iii. New WordCount Project

file- > Project , select Map/reduce Project, enter the name WordCount, and so on.

Create a new class in the WordCount project named WordCount with the following code:

Importjava.io.IOException;ImportJava.util.StringTokenizer;Importorg.apache.hadoop.conf.Configuration;ImportOrg.apache.hadoop.fs.Path;Importorg.apache.hadoop.io.IntWritable;ImportOrg.apache.hadoop.io.Text;ImportOrg.apache.hadoop.mapreduce.Job;ImportOrg.apache.hadoop.mapreduce.Mapper;ImportOrg.apache.hadoop.mapreduce.Reducer;ImportOrg.apache.hadoop.mapreduce.lib.input.FileInputFormat;ImportOrg.apache.hadoop.mapreduce.lib.output.FileOutputFormat;ImportOrg.apache.hadoop.util.GenericOptionsParser; Public classWordCount { Public Static classTokenizermapperextendsMapper<object, text, text, intwritable>{Private Final StaticIntwritable one =NewIntwritable (1); PrivateText Word =NewText ();  Public voidmap (Object key, Text value, context context)throwsIOException, interruptedexception {stringtokenizer ITR=NewStringTokenizer (value.tostring ());  while(Itr.hasmoretokens ()) {Word.set (Itr.nexttoken ());　　　　　　Context.write (Word, one); }}} Public Static classIntsumreducerextendsReducer<text,intwritable,text,intwritable>{Privateintwritable result =Newintwritable ();  Public voidReduce (Text key, iterable<intwritable>values,context Context)throwsIOException, interruptedexception {intsum = 0;  for(intwritable val:values) {sum+=Val.get ();　　　　} result.set (sum);　　Context.write (key, result); }}  Public Static voidMain (string[] args)throwsException {Configuration conf=NewConfiguration (); String[] Otherargs=Newgenericoptionsparser (conf, args). Getremainingargs (); if(Otherargs.length! = 2) {System.err.println ("Usage:wordcount <in> <out>"); System.exit (2); } Job Job=NewJob (conf, "word count"); Job.setjarbyclass (WordCount.class); Job.setmapperclass (tokenizermapper.class); Job.setcombinerclass (intsumreducer.class); Job.setreducerclass (intsumreducer.class); Job.setoutputkeyclass (Text.class); Job.setoutputvalueclass (intwritable.class); Fileinputformat.addinputpath (Job,NewPath (otherargs[0])); Fileoutputformat.setoutputpath (Job,NewPath (otherargs[1])); System.exit (Job.waitforcompletion (true) ? 0:1);}}

Four, the Operation

1. Create a directory on HDFS input

Hadoop fs-mkdir Input

2. Copy the local README.txt to the input in HDFs

Hadoop fs-copyfromlocal/usr/local/hadoop/readme.txt Input

3, click Wordcount.java, right click on Run As->run configurations, configure the run parameters, namely the input and output folder

Hdfs://localhost:9000/user/hadoop/input Hdfs://localhost:9000/user/hadoop/output

Click the Run button to run the program.

4, after the completion of the operation, view the results of the operation

Method 1:

Hadoop fs-ls Output

you can see that there are two output results, _success and part-r-00000

perform Hadoop fs-cat output/*

Method 2:

Expand Dfs Locations, as shown, double-click Open part-r00000 View Results

Hadoop Eclipse Development Environment Building