hadoop-2.2.0 Configuring the Eclipse Plug-in (Windows and Linux platforms)

There are currently two versions of the Eclipse plug-in configured, one under Windows recompile, and the other on Linux in reconfiguration.

Step through the process of compiling-configuring the Eclipse plug-in under Linux.

2. environment:

    Ubuntu 12.04 64bit 3.2.0-29-generic eclipse-jee-luna-r-linux-gtk-x86_64.tar.gz Apache Ant (TM) version 1.8.2 JDK version 1.7.0_67  

4. Preparation before installation:

   Ant, JDK, Eclipse, Apache Hadoop 2.2.0 installation packages are ready.
   Eclipse Plugin Source code ready (Https://github.com/winghc/hadoop2x-eclipse-plugin)
   Deploy Apache Hadoop 2.2.0 on the cluster.

5. To compile the plugin:

   • Copy the Apache Hadoop 2.2.0 installation package and Eclipse to any directory, such as/home/hadoop/downloads/hadoop-2.2.0 and/home/hadoop/downloads/eclipse. Remember: The entire path can not appear any space, or the ant compilation process will error!

   • Enter the plugin directory to compile, such as plugins placed in/home/hadoop/downloads/hadoop2x-eclipse-plugin

     $ cd /home/hadoop/Downloads/hadoop2x-eclipse-plugin/src/contrib/eclipse-plugin$ ant jar -Dversion=2.2.0 -Declipse.home=/home/hadoop/Downloads/eclipse -Dhadoop.home=/home/hadoop/Downloads/hadoop-2.2.0

     Note: where-declipse.home and-dhadoop.home need to specify the storage directory for the installation directory and Hadoop installation files for Eclipse, remember: spaces cannot exist in the full path of both directories

     The next is the long wait, the main slow is target:ivy-download,ivy-resolve-common these two steps.
     The last plugin generated is:
     Hadoop2x-eclipse-plugin/build/contrib/eclipse-plugin/hadoop-eclipse-plugin-2.2.0.jar

6. Next plug-in installation and configuration

   • Copy Hadoop-eclipse-plugin-2.2.0.jar to eclipse's plugins directory and start eclipse

   • Enter the window-->preference configuration root directory:
     
     

   • Enter Window-->open perspective-->other-->map/reduce open map/reduce window
     
     

   • Open Map/reduce View, right-click, selectNew hadoop location
     
     

     Configure location

     Note: The MR master and DFS master configurations must be consistent with the configuration files such as Mapred-site.xml and Core-site.xml

   • Open Project Explorer to view the HDFs file system:
     
     

   • New MapReduce Project
     
     

   • Create a MapReduce program

7. Write the WordCount program test plug-in using:

   • Create a new Wordcount.java program under the MapReduce project with the following code:

     Import Java.io.ioexception;import java.util.*;import org.apache.hadoop.fs.path;import org.apache.hadoop.conf.*; Import org.apache.hadoop.io.*;import org.apache.hadoop.mapred.*;import org.apache.hadoop.util.*;p ublic class WordCount {public static class Map extends Mapreducebase implements Mapper<longwritable, text, text, intwritable> {p Rivate final static intwritable one = new intwritable (1);p rivate text word = new Text ();p ublic void map (longwritable key, Text value, Outputcollector<text, intwritable> output, Reporter Reporter) throws IOException {String line = value.t Ostring (); StringTokenizer tokenizer = new StringTokenizer (line);   while (Tokenizer.hasmoretokens ()) {Word.set (Tokenizer.nexttoken ()); Output.collect (Word, one); }}}public Static class Reduce extends Mapreducebase implements Reducer<text, Intwritable, Text, intwritable> {publi c void reduce (Text key, iterator<intwritable> values, Outputcollector<text, intwritable> output, Reporter Reporter throws IOException {int sum = 0, while (Values.hasnext ()) {sum + = Values.next (). get ();} output.collect (Key, New INTW Ritable (sum));}} public static void Main (string[] args) throws Exception {jobconf conf = new jobconf (wordcount.class); Conf.setjobname ("Wor DCount "); Conf.setoutputkeyclass (Text.class); Conf.setoutputvalueclass (Intwritable.class); Conf.setmapperclass ( Map.class); Conf.setreducerclass (Reduce.class); Conf.setinputformat (Textinputformat.class); Conf.setOutputFormat ( Textoutputformat.class); Fileinputformat.setinputpaths (conf, new Path (Args[0])); Fileoutputformat.setoutputpath (conf, new Path (Args[1])); Jobclient.runjob (conf);}}

   • Configure run-time parameters: Right-click-->run as-->run confiugrations
     
     

   • Run the program on a Hadoop cluster: Right--->runas-->run on Hadoop, and the final output will be displayed in the appropriate folder in HDFs. At this point, the Linux hadoop-2.2.0 Eclipse plugin configuration is complete.
     
     

8. The first problem in the configuration process:

   • An issue that cannot be written to the file HDFs file system in Eclipse, which will directly cause programs written under eclipse not to run on Hadoop.

     There are three solutions on the Web:
     A. Remove user permission checks for Hadoop HDFs. Open Conf/hdfs-site.xml, locate the Dfs.permissions property, and modify it to False (default is True). [My original configuration file was written in Dfs.permission, although this value changed to False, but still invalid, resulting in this environment configuration is not successful, tears ran]

     B. To modify the Hadoop location parameter, in the Advanced Parameter tab, locate the Hadoop.job.ugi option and change this option to start the Hadoop username (Note that The first time you set the time may not have Hadoop.job.ugi parameters, after the error to see there is a) [in fact, I did not find this option]

     C. Because eclipse submits jobs by using Hadoop plug-ins, the jobs are written to the HDFs file system by default as Drwho, and the corresponding/usr/hadoop on HDFs, because drwho users do not have write access to the Hadoop directory, Therefore causes an exception to occur. The workaround is: Release the Hadoop directory permissions, the command is as follows: $hadoop fs -chmod 777 [at the time of configuration, only release the permissions of a few directories, so that the job could not commit to write to HDFs, cried dead ...]

   • In addition, the use of Eclipse plug-ins may be related to the Eclipse version, I am using the eclipse-jee-luna-r-linux-gtk-x86_64.tar.gz version of Eclipse, in Ubuntu 12.04 Can be used under 64bit

Here are some examples of how to configure the Eclipse plugin under Windows compilation

2. environment:

    Windows 7 64bit Professional Edition eclipse-standard-luna-r-win32-x86_64 Apache Ant (TM) version 1.9.4 JDK Version 1.7.0_67  

4. Preparation before installation:

   Ant, JDK, Eclipse, Apache Hadoop 2.2.0 installation packages are ready.
   Eclipse Plugin Source code ready (Https://github.com/winghc/hadoop2x-eclipse-plugin)
   Deploy Apache Hadoop 2.2.0 on the cluster.

5. To compile the plugin:

   • Copy the Apache Hadoop 2.2.0 installation package and Eclipse to any directory, such as E:\hadoop-2.2.0 and E:\eclipse. Remember: The entire path can not appear any space, or the ant compilation process will error!

   • Enter the plugin directory to compile, such as plugins placed in E:\hadoop2x-eclipse-plugin

     $ cd hadoop2x-eclipse-plugin/src/contrib/eclipse-plugin$ ant jar -Dversion=2.2.0 -Declipse.home=E:\eclipse -Dhadoop.home=E:\hadoop-2.2.0

     Note: where-declipse.home and-dhadoop.home need to specify the storage directory for the installation directory and Hadoop installation files for Eclipse, remember: spaces cannot exist in the full path of both directories

     The next is the long wait, the main slow is target:ivy-download,ivy-resolve-common these two steps.
     The last plugin generated is:
     Hadoop2x-eclipse-plugin/build/contrib/eclipse-plugin/hadoop-eclipse-plugin-2.2.0.jar

6. Configure the Hadoop environment variable in window 7:

   • Set the%hadoop_home% in Window7 and add%hadoop_home%\bin to the PATH environment variable [This step if not done, the HADOOP program runs the process of a null pointer exception, so that the program fails to run]

   • Also need to download a plugin https://github.com/srccodes/hadoop-common-2.2.0-bin, unzip the downloaded bin directory to overwrite the files in the%hadoop_home%\bin directory

     Note that these two steps are necessary, otherwise it will lead to a variety of wonderful mistakes

     The next steps are the same as in the 4, 5 steps in the Linux version configuration scenario

7. Issues that occur during configuration:

   • When using ant to compile the eclipse plugin , a bunch of warnings appear, such as

     [Javac] E:\hadoop-2.2.0\share\hadoop\hdfs\hadoop-hdfs-2.2.0.jar (org/apache/hadoop/hdfs/distributedfilesystem.class): Warning : Unable to find comment method for type ' Limitedprivate ' ' value () ' [Javac] E:\hadoop-2.2.0\share\hadoop\common\hadoop-common-2.2.0.jar (org/ Apache/hadoop/fs/filesystem.class): Warning: Unable to find comment method of type ' Limitedprivate ' ' value () ' [Javac] E:\hadoop-2.2.0\share\hadoop\ Common\hadoop-common-2.2.0.jar (Org/apache/hadoop/fs/filesystem.class): Warning: Unable to find comment method for type ' Limitedprivate ' ' value () ' [ Javac] E:\hadoop-2.2.0\share\hadoop\common\hadoop-common-2.2.0.jar (org/apache/hadoop/fs/filesystem.class): Warning: Unable to find comment method for type ' Limitedprivate ' ' value () ' [Javac] E:\hadoop-2.2.0\share\hadoop\common\hadoop-common-2.2.0.jar (org/ Apache/hadoop/fs/fsdatainputstream.class): Warning: Unable to find comment method of type ' Limitedprivate ' ' value () ' [Javac] E:\hadoop-2.2.0\share\ Hadoop\common\hadoop-common-2.2.0.jar (Org/apache/hadoop/fs/fsdataoutputstream.class): Warning: Unable to find type ' LimitedPrivate ' Annotation method ' value () ' [Javac] Note: Some input files use or overwrite outdated APIs. [Javac] Note: For more information, please use-xlint:deprecatIon recompile. [Javac] Note: Some input files use an unchecked or unsafe operation. [Javac] Note: For more information, please use-xlint:unchecked to recompile. [Javac] 7 warnings

     But eventually it compiles successfully, and these warnings do not need to be managed.

   • The installation of the plug-in was successful with the Eclipse version, and I used the Windows 64bit Java EE version of Eclipse, which was compiled with the plugin one through Ant and found that eclipse does not recognize the plugin.

   • In addition, in the source code of the Eclipse plugin, there is an already compiled Eclipse plugin, which is not available and is used in eclipse, and will be reported classnotdefexception

hadoop-2.2.0 Configuring the Eclipse Plug-in (Windows and Linux platforms)