MapReduce program Local mode debugging

No jar version Windows platform Hadoop-2.6.1.zip

Eclipse

Extract

Conf.set ("Mapreduce.framework.name", "local");
Conf.set ("Fs.defaultfs", "file:///");

Run parameters

The result was a successful run.

Conf.set ("Mapreduce.framework.name", "local");
System.setproperty ("Hadoop_user_name", "HADOOP");
Conf.set ("Fs.defaultfs", "hdfs://hadoop01:9000");

The result was a successful run.

4.1 Error org.apache.hadoop.security.AccessControlException:Permission Denied:user=admin, Access=write, inode= "/ WordCount ": hadoop:supergroup:drwxr-xr-x

Not add System.setproperty ("Hadoop_user_name", "HADOOP");

4.2Exception in thread ' main ' java.lang.illegalargumentexception:pathname/c:/wordcount/output6666 from hdfs:// hadoop01:9000/c:/wordcount/output6666 is not a valid DFS filename

(1) The MapReduce program is to be submitted to the Localjobrunner in the form of a single process

(2) The processing of data and output can be in the local file system, can also be on the HDFs

(3) How to achieve local operation? Write a program that does not have a clustered configuration file (essentially whether there are mapreduce.framework.name=local and Yarn.resourcemanager.hostname parameters in the Conf of your Mr Program)

(4) Local mode is very easy to debug business logic, as long as you interrupt in eclipse

If you want to run local mode to test program logic under Windows, you need to configure the environment variables in Windows:

%hadoop_home% = d:/hadoop-2.6.1

%PATH% =%hadoop_home%\bin

And to replace the d:/hadoop-2.6.1 Lib and bin directories with the version of the Windows platform compilation