Issues encountered by eclipse submitting tasks to the Hadoop cluster

Environment: Windows8.1,eclipse

WordCount example with Hadoop comes in

hadoop2.7.0

Hadoop-eclipse-plugin-2.7.0.jar//eclipse plugin, need to correspond to the current version of Hadoop

Basic steps there are a lot of blogs that have been mentioned and don't repeat

1. Put Hadoop-eclipse-plugin-2.7.0.jar into Eclipse's plugins directory to launch eclipse

2. Configure the Hadoop location information for Eclipse

3. New MapReduce Project

4. Copy the WordCount code in

/** * Licensed to the Apache software Foundation (ASF) under one * or more contributor license agreements.  See the NOTICE file * Distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * under the Apache License, Version 2.0 (The * "License");  You are not a use of this file except in compliance * with the License. Obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * unless required by applicab Le law or agreed into writing, software * Distributed under the License is distributed on a "as is" BASIS, * without WAR Ranties or CONDITIONS of any KIND, either express OR implied. * See the License for the specific language governing permissions and * limitations under the License. */package Org.apache.hadoop.examples;import Java.io.ioexception;import Java.util.stringtokenizer;import Org.apache.hadoop.conf.configuration;import Org.apache.hadoop.fs.path;import Org.apache.hadoop.io.IntWritablE;import Org.apache.hadoop.io.text;import Org.apache.hadoop.mapreduce.job;import Org.apache.hadoop.mapreduce.mapper;import Org.apache.hadoop.mapreduce.reducer;import Org.apache.hadoop.mapreduce.lib.input.fileinputformat;import Org.apache.hadoop.mapreduce.lib.output.fileoutputformat;import Org.apache.hadoop.util.GenericOptionsParser; public class WordCount {public static class Tokenizermapper extends Mapper<object, text, text, intwritable>    {Private final static intwritable one = new intwritable (1);          Private text Word = new text ();      public void Map (Object key, Text value, Context context) throws IOException, Interruptedexception {      StringTokenizer ITR = new StringTokenizer (value.tostring ());        while (Itr.hasmoretokens ()) {Word.set (Itr.nexttoken ());      Context.write (Word, one); }}} public static class Intsumreducer extends Reducer<text,intwritable,text,intwritable> {privat E intwritable ResUlt = new Intwritable ();                       public void reduce (Text key, iterable<intwritable> values, context context      ) throws IOException, interruptedexception {int sum = 0;      for (intwritable val:values) {sum + = Val.get ();      } result.set (sum);    Context.write (key, result);   }} public static void Main (string[] args) throws Exception {Configuration conf = new configuration (); Conf.set ("Mapred.job.tracker", "192.168.1.150:9001"); Conf.set ("Yarn.resourcemanager.address", "        192.168.1.150:8032 ");    string[] Otherargs = new Genericoptionsparser (conf, args). Getremainingargs (); if (Otherargs.length < 2) {System.err.println ("Usage:wordcount <in> [<in> ...]      <out> ");    System.exit (2);    Job Job = job.getinstance (conf, "word count");    Job.setjarbyclass (Wordcount.class);    Job.setmapperclass (Tokenizermapper.class);    Job.setcombinerclass (Intsumreducer.class); Job.setreducerClass (Intsumreducer.class);    Job.setoutputkeyclass (Text.class);    Job.setoutputvalueclass (Intwritable.class);    for (int i = 0; i < otherargs.length-1; ++i) {Fileinputformat.addinputpath (Job, New Path (Otherargs[i]));    } fileoutputformat.setoutputpath (Job, New Path (Otherargs[otherargs.length-1]));  System.exit (Job.waitforcompletion (true)? 0:1); }}

　　

The first three lines of code for the main method need to be configured yourself

5. Copy the configuration files from the deployed Hadoop cluster to the project

Log4j.properties must be configured, otherwise the console cannot display the information when the task is submitted to the cluster, the following is my configuration

Log4j.rootlogger=debug, calog4j.appender.ca=org.apache.log4j.consoleappenderlog4j.appender.ca.layout= org.apache.log4j.patternlayoutlog4j.appender.ca.layout.conversionpattern=%-4r [%t]%-5p%c%x-%m%n

　　

6. Right click on Run as--Wordcount.java

Error 1:

Org.apache.hadoop.util.shell$exitcodeexception:/bin/bash:line 0:fg:no Job Control

Problems raised when Hadoop reads Windows and Linux system variables, there are several solutions that bother not to recompile the entire Hadoop directly in this project to resolve

Find the Yarnrunner.java in the source code of Hadoop, copy it to the project, and the package in the project will be the same as in the Hadoop source, and the runtime will overwrite

Modify Yarnrunner.java

(1) Modify how Windows system variables are read

The commented out code is the original code.

(2) Add a new way to process Windows system variables

  private void Replaceenvironment (map<string, string> environment) {      String Tmpclasspath = Environment.get (" CLASSPATH ");      Tmpclasspath=tmpclasspath.replaceall (";", ":");      Tmpclasspath=tmpclasspath.replaceall ("%pwd%", "\ \ $PWD");      Tmpclasspath=tmpclasspath.replaceall ("%hadoop_mapred_home%", "\ \ $HADOOP _mapred_home");      Tmpclasspath= Tmpclasspath.replaceall ("\\\\", "/");      Environment.put ("CLASSPATH", Tmpclasspath);}

Use this place

Error 2:

Exited with ExitCode: 1 due to:exception from Container-launch

Diagnostics:exception from Container-launch.

Modify the Mapred-site.xml in the project to add the following:

< Property> <name>Mapreduce.application.classpath</name> <value>$HADOOP _conf_dir, $HADOOP _common_home/share/hadoop/common/*, $HADOOP _common_home/share/hadoop/common/lib/*, $ hadoop_hdfs_home/share/hadoop/hdfs/*, $HADOOP _hdfs_home/share/hadoop/hdfs/lib/*, $HADOOP _mapred_home/share/ hadoop/mapreduce/*, $HADOOP _mapred_home/share/hadoop/mapreduce/lib/*, $HADOOP _yarn_home/share/hadoop/yarn/*, $ hadoop_yarn_home/share/hadoop/yarn/lib/*</value> </ Property>

Viewcode

Issues encountered by eclipse submitting tasks to the Hadoop cluster