Windows compiled Hadoop 2.x Hadoop-eclipse-plugin plugin

A. Introduction

Without the Eclipse plugin tool after hadoop2.x, we can't debug the code on eclipse, we're going to package the MapReduce of the written Java code into a jar and run it on Linux, so it's inconvenient for us to debug the code, so we compile an eclipse plugin ourselves, so we can easily We debug in our local, after hadoop1.x development, compiling the hadoop2.x version of the Eclipse plugin is much simpler than before. Next we started compiling the Hadoop-eclipse-plugin plugin and developing Hadoop in eclipse.

two. Software Installation and Configuration

1.JDK Configuration

1) Installing the JDK

2) Configure Environment variables

Java_home, CLASSPATH, path and other settings, here is not much introduction, online a lot of information

2.Eclipse

1). Download Eclipse-jee-juno-sr2.rar

2). Unzip to the local disk:

3.Ant

1) Download

http://ant.apache.org/bindownload.cgi

Apache-ant-1.9.4-bin.zip

2) Extract to a disk:

3). Configuration of environment variables

New ant_home=e:\ant\apache-ant-1.9.4-bin\apache-ant-1.9.4

Add to the back of path;%ant_home%\bin

4) CMD Test if the configuration is correct

Ant version:

4.Hadoop

1). Download the Hadoop package

Hadoop-2.6.0.tar.gz

Unzip to the local disk:

Download Hadoop2x-eclipse-plugin Source code

1) The current HADOOP2 eclipse-plugins source code is de-administered by GitHub, is Https://github.com/winghc/hadoop2x-eclipse-plugin, and then on the right side of the download Zip connection click to download:

2) Download Hadoop2x-eclipse-plugin-master.zip

Unzip to the local disk:

three. Compiling the Hadoop-eclipse-plugin plugin

1.hadoop2x-eclipse-plugin-master Decompression on the e: Disk open command line cmd, switch to E:\hadoop\hadoop2x-eclipse-plugin-master\src\contrib\ Eclipse-plugin directory,:

2. Execute the ant jar

Antjar-dversion=2.6.0-declipse.home=F:\tool\eclipse-jee-juno-SR2\eclipse-jee-juno-SR2 -dhadoop.home= E:\hadoop\hadoop-2.6.0\hadoop-2.6.0,:

3. Compile successfully generated Hadoop-eclipse-plugin-2.6.0.jar in E:\hadoop\hadoop2x-eclipse-plugin-master\build\contrib\ Eclipse-plugin Path, the following:   

Four. Eclipse Configuration Hadoop-eclipse-plugin Plugin   1. Copy the Hadoop-eclipse-plugin-2.6.0.jar to the F:\tool\eclipse-jee-juno-SR2\eclipse-jee-juno-SR2\plugins directory, Restart Eclipse, and then you can see the DFS Locations:

2. Open Window-->preferens, you can see the Hadoop map/reduc option, then click, then add hadoop-2.6.0 come in,:

3. Configure Map/reducelocations

1) Click Window-->show View-->mapreduce Tools to click Map/reducelocation

2) Click on the Map/reducelocation tab and click on the icon on the right to open the Hadoop Location Configuration window: Enter location name and any name. Configure Map/reduce Master and DFS Mastrer, Host and Port are configured to Hdfs-site.xml with Core-site.xml settings.

4. See if the connection is successful

Five. Run the new WordCount project and run

1. Right-click New->map/reduce Project

2. New Wordcount.java

Import Java.io.ioexception;import Java.util.stringtokenizer;import Org.apache.hadoop.conf.configuration;import Org.apache.hadoop.fs.path;import Org.apache.hadoop.io.intwritable;import Org.apache.hadoop.io.text;import Org.apache.hadoop.mapreduce.job;import Org.apache.hadoop.mapreduce.mapper;import Org.apache.hadoop.mapreduce.reducer;import Org.apache.hadoop.mapreduce.lib.input.fileinputformat;import       Org.apache.hadoop.mapreduce.lib.output.fileoutputformat;public class WordCount {public static class Tokenizermapper    Extends Mapper<object, text, text, intwritable>{private final static intwritable one = new intwritable (1);    Private text Word = new text ();      public void Map (Object key, Text value, Context context) throws IOException, Interruptedexception {      StringTokenizer ITR = new StringTokenizer (value.tostring ());        while (Itr.hasmoretokens ()) {Word.set (Itr.nexttoken ());      Context.write (Word, one); }}} public static CLass Intsumreducer extends reducer<text,intwritable,text,intwritable> {private intwritable result = new in    Twritable ();                       public void reduce (Text key, iterable<intwritable> values, context context      ) throws IOException, interruptedexception {int sum = 0;      for (intwritable val:values) {sum + = Val.get ();      } result.set (sum);    Context.write (key, result);    }} public static void Main (string[] args) throws Exception {Configuration conf = new configuration ();    Job Job = job.getinstance (conf, "word count");    Job.setjarbyclass (Wordcount.class);    Job.setmapperclass (Tokenizermapper.class);    Job.setcombinerclass (Intsumreducer.class);    Job.setreducerclass (Intsumreducer.class);    Job.setoutputkeyclass (Text.class);    Job.setoutputvalueclass (Intwritable.class);    Fileinputformat.addinputpath (Job, New Path (Args[0]));    Fileoutputformat.setoutputpath (Job, New Path (Args[1])); System.Exit (Job.waitforcompletion (TRUE)? 0:1); }}

3. Create a text to be counted in the HDFs input directory

1) No input/output directory card, first build a folder on HDFs

#bin/hdfs Dfs-mkdir–p/user/root/input

#bin/hdfs dfs-mkdir-p/user/root/output

2). Upload the text you want to count to the input directory of HDFs

# Bin/hdfs dfs-put/usr/local/hadoop/hadoop-2.6.0/test/*/user/root/input//upload tes/file01 file to HDFs/user/root/input

3). View

#bin/hdfs dfs-cat/user/root/input/file01

4. Click Wordcount.java Right-click-->run as-->run Configurations set the input and output directory path:

5. Click Wordcount.java Right-click-->run as-->run on Hadoop

Then to the Output/count directory, there is a statistic file and view the results, so the configuration is successful.

Five. Place of attention

In this article, Eclipse connects to Hadoop on a Linux virtual machine and develops hadoop issues in eclipse, solving exception:org.apache.hadoop.io.nativeio.nativeio$ WINDOWS.ACCESS0 (Ljava/lang/string;i) Z and a series of questions

Windows compiled Hadoop 2.x Hadoop-eclipse-plugin plugin