Eclipse Connection Hadoop2.7.3 cluster configuration and testing

1. Install the Hadoop2.7.3 6 node cluster and HA with QJM and RM

Node	Nn	Dn	Zkfc	Zk	JN	Rm	NM
Master	1		1	1	1	1
Salve1	1		1	1	1	1
Salve2		1		1	1		1
Salve3		1					1
Salve4		1					1
Salve5		1					1

2. Compile hadoop-eclipse-plugin-2.7.3 based on Hadoop2x-eclipse-plugin Https://github.com/winghc/hadoop2x-eclipse-plugin 3. Eclipse version:eclipse-java-luna-sr2-win32-x86_64 4.  Useful hadoop2.7.3 Hadoop.dll and Winutils.exe.zip files 5. Unzip the hadoop-2.7.3.tar.gz to the Windows directory and put the extracted Bin folder in step 4 into the Hadoop bin directory under Windows 6. Install Eclipse and put the compiled hadoop-eclipse-plugin-2.7.3 into the plugin directory Eclipse configuration as shown below
7. Ensure that master active in the cluster and connect to the Hadoop cluster, the following connection is successful
8. We can now access Hadoop cluster HDFs files through eclipse in Windows, but the eclipse development WordCount still cannot submit and view HDFs results in Windows. Need some additional configuration as described below to achieve 9. Compiling Eclipse wordcount Code

Package z_stu;
Import java.io.IOException;
Import Java.util.StringTokenizer;
Import org.apache.hadoop.conf.Configuration;
Import Org.apache.hadoop.fs.Path;
Import org.apache.hadoop.io.IntWritable;
Import Org.apache.hadoop.io.Text;
Import Org.apache.hadoop.mapreduce.Job;
Import Org.apache.hadoop.mapreduce.Mapper;
Import Org.apache.hadoop.mapreduce.Reducer;
Import Org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
Import Org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class WordCount {public static class Tokenizermapper extends Mapper<object, text, text, intwritable> {Priva
Te final static intwritable one = new intwritable (1);
Private text Word = new text (); public void Map (Object key, Text value, Context context) throws IOException, interruptedexception {StringTokenizer ITR =
New StringTokenizer (Value.tostring ());
while (Itr.hasmoretokens ()) {Word.set (Itr.nexttoken ()); Context.write (Word, one);}} } public static class Intsumreducer extends Reducer<text, intwritable, text, intwritable> {private intwritable result = new intwritable (); public void reduce (Text key, Iterabl  E<intwritable> values, Context context) throws IOException, interruptedexception {int sum = 0; for (intwritable Val
: values) {sum + = Val.get ();} result.set (sum);
Context.write (key, result);
}} public static void Main (string[] args) throws Exception {Configuration conf = new configuration ();
Job Job = job.getinstance (conf, "word count");
Job.setjarbyclass (Wordcount.class);
Job.setmapperclass (Tokenizermapper.class);
Job.setcombinerclass (Intsumreducer.class);
Job.setreducerclass (Intsumreducer.class);
Job.setoutputkeyclass (Text.class);
Job.setoutputvalueclass (Intwritable.class);
Fileinputformat.addinputpath (Job, New Path (Args[0]));
Fileoutputformat.setoutputpath (Job, New Path (Args[1]));
System.exit (Job.waitforcompletion (true)? 0:1); }
}

10. Import a running cluster etc directory 5 files and modify mapred-site.xml/mapred-site.xml see figure below


11. Configuring the Windows Hosts file
12. Configure Windows user and system environment variables

13. Import the Worldcount jar into Eclipse 14. Run
15. View results in Hdfs/eclipse