Eclipse串連Hadoop2.7.3叢集配置及測試

1. 安裝Hadoop2.7.3 6節點叢集並HA with QJM and RM

node	NN	DN	ZKFC	ZK	JN	RM	NM
master	1		1	1	1	1
salve1	1		1	1	1	1
salve2		1		1	1		1
salve3		1					1
salve4		1					1
salve5		1					1

2. 編譯hadoop-eclipse-plugin-2.7.3基於hadoop2x-eclipse-plugin https://github.com/winghc/hadoop2x-eclipse-plugin 3. Eclipse version: eclipse-java-luna-SR2-win32-x86_64 4. 有用的hadoop2.7.3 hadoop.dll和winutils.exe.zip檔案 5.  解壓hadoop-2.7.3.tar.gz到windows目錄並把步驟4解壓後的bin檔案夾放到windows下hadoop bin目錄中 6. 安裝eclipse並放編譯好的hadoop-eclipse-plugin-2.7.3進plugin目錄 Eclipse 配置如下圖
7. 確保叢集中master active 並串連hadoop叢集，如下串連成功
8. 現在我們可以通過eclipse在windows訪問hadoop叢集hdfs檔案，但是eclipse開發wordcount時仍然不能在windows提交並查看hdfs結果。需要一些額外配置詳見如下來實現 9. 編譯eclipse wordcount代碼

package z_stu;import java.io.IOException;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;public class WordCount {public static class TokenizerMapper extendsMapper<Object, Text, Text, IntWritable> {private final static IntWritable one = new IntWritable(1);private Text word = new Text();public void map(Object key, Text value, Context context)throws IOException, InterruptedException {StringTokenizer itr = new StringTokenizer(value.toString());while (itr.hasMoreTokens()) {word.set(itr.nextToken());context.write(word, one);}}}public static class IntSumReducer extendsReducer<Text, IntWritable, Text, IntWritable> {private IntWritable result = new IntWritable();public void reduce(Text key, Iterable<IntWritable> values,Context context) throws IOException, InterruptedException {int sum = 0;for (IntWritable val : values) {sum += val.get();}result.set(sum);context.write(key, result);}}public static void main(String[] args) throws Exception {Configuration conf = new Configuration();Job job = Job.getInstance(conf, "word count");job.setJarByClass(WordCount.class);job.setMapperClass(TokenizerMapper.class);job.setCombinerClass(IntSumReducer.class);job.setReducerClass(IntSumReducer.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);FileInputFormat.addInputPath(job, new Path(args[0]));FileOutputFormat.setOutputPath(job, new Path(args[1]));System.exit(job.waitForCompletion(true) ? 0 : 1);}}

10. 匯入已運行叢集etc目錄5個檔案及修改mapred-site.xml/mapred-site.xml見下圖


11. 配置windows hosts檔案
12. 配置windows使用者及系統內容變數

13. 把worldcount jar匯入eclipse 14. 運行
15. 在hdfs/eclipse查看結果