Configure the Hadoop MapReduce development environment in Eclipse

Environment:

Eclipse version: MyEclipse6.5.1

Hadoop version: hadoop-1.2.1

 

1. After installing MyEclipse, create a java Project

File-> New-> Java Project

Enter the project name and click OK.

 

2. Import all hadoop packages

Decompress hadoop-1.2.1.tar (E: \ software \ share \ hadoop-1.2.1)

Put E: \ software \ share \ hadoop-1.2.1 under

And E: \ software \ share \ hadoop-1.2.1 \ lib under the jar package are imported into the project

The method is as follows:

Right-click the project root and choose Properties> JavaPath> Libraries> Add External JARs.

 

3. confirm that the jre version is 6.0 or later.

My MyEclipse6.5.1 version started to use jre5.0 by default, because the hadoop-1.2.1 needs jre 6.0 or later versions, the execution of the Program reported an error:

Bad version number in. class file (unableto load class ***)

 

How to change the jre version

Windows-> Preference-> Java-> InstalledJREs à add

 

4. Modify the FileUtil. java File

At this time, when creating a mapreduce program to test WordCount, we also encountered the following problems:

 

13/12/13 22:58:49 WARNutil. NativeCodeLoader: Unable to load native-hadoop library for yourplatform... using builtin-java classes where applicable

13/12/13 22:58:49 ERRORsecurity. userGroupInformation: PriviledgedActionExceptionas: liczcause: java. io. IOException: Failed to set permissions of path: \ tmp \ hadoop-licz \ mapred \ staging \ licz1853694772 \. staging to 0700

Exception in thread "main" java. io. IOException: Failed to set permissions of path: \ tmp \ hadoop-licz \ mapred \ staging \ licz1853694772 \. staging

......

 

Solution:

Modify E: \ software \ share \ hadoop-1.2.1 \ src \ core \ org \ apache \ hadoop \ fs \FileUtil. java File

Comment the following content

 

685 private static voidcheckReturnValue (boolean rv, File p,

686 FsPermission permission

(687) throws IOException {

688/* if (! Rv ){

689 throw new IOException ("Failed toset permissions of path:" + p +

690 "to" +

691 String. format ("% 04o", permission. toShort ()));

692 }*/

693}

 

Create an org. apache. hadoop. fs package in Mapreduce1/scr and copy the FileUtil. java file to the package (paste it in eclipse)

 

No error is reported when the WordCount. java program is compiled again.

Import java. io. IOException;
Import java. util. Iterator;
Import java. util. StringTokenizer;

Import org. apache. hadoop. fs. Path;
Import org. apache. hadoop. io. IntWritable;
Import org. apache. hadoop. io. Text;
Import org. apache. hadoop. mapred. FileInputFormat;
Importorg. apache. hadoop. mapred. FileOutputFormat;
Importorg. apache. hadoop. mapred. JobClient;
Importorg. apache. hadoop. mapred. JobConf;
Import org. apache. hadoop. mapred. MapReduceBase;
Importorg. apache. hadoop. mapred. Mapper;
Importorg. apache. hadoop. mapred. OutputCollector;
Importorg. apache. hadoop. mapred. Cer CER;
Importorg. apache. hadoop. mapred. Reporter;
Importorg. apache. hadoop. mapred. TextInputFormat;
Importorg. apache. hadoop. mapred. TextOutputFormat;

Public class WordCount {

Public static class WordCountMapper extends MapReduceBase implementsMapper <Object, Text, Text, IntWritable> {
Private final static IntWritable one = new IntWritable (1 );
Private Text word = new Text ();

Public void map (Object key, Text value, OutputCollector <Text, IntWritable> output, Reporter reporter) throws IOException {
StringTokenizer itr = newStringTokenizer (value. toString ());
While (itr. hasMoreTokens ()){
Word. set (itr. nextToken ());
Output. collect (word, one );
}

}
}

Public static class WordCountReducer extends MapReduceBase implementsReducer <Text, IntWritable, Text, IntWritable> {
Private IntWritable result = new IntWritable ();

Public void reduce (Text key, Iterator <IntWritable> values, OutputCollector <Text, IntWritable> output, Reporter reporter) throws IOException {
Int sum = 0;
While (values. hasNext ()){
Sum + = values. next (). get ();
}
Result. set (sum );
Output. collect (key, result );
}

}

Public static void main (String [] args) throws Exception {
String input = "hdfs: // 192.168.2.100: 9000/user/licz/hdfs/o_t_account ";
String output = "hdfs: // 192.168.2.100: 9000/user/licz/hdfs/o_t_account/result ";

JobConf conf = new JobConf (WordCount. class );
Conf. setJobName ("WordCount ");
Conf. addResource ("classpath:/hadoop/core-site.xml ");
Conf. addResource ("classpath:/hadoop/hdfs-site.xml ");
Conf. addResource ("classpath:/hadoop/mapred-site.xml ");

Conf. setOutputKeyClass (Text. class );
Conf. setOutputValueClass (IntWritable. class );

Conf. setMapperClass (WordCountMapper. class );
Conf. setCombinerClass (WordCountReducer. class );
Conf. setReducerClass (WordCountReducer. class );

Conf. setInputFormat (TextInputFormat. class );
Conf. setOutputFormat (TextOutputFormat. class );

FileInputFormat. setInputPaths (conf, new Path (input ));
FileOutputFormat. setOutputPath (conf, new Path (output ));

JobClient. runJob (conf );
System. exit (0 );
}

}

Note:

If you use eclipse on windows, the user name must be the same as the user name for hadoop to be installed on the hadoop server. Otherwise, an error occurs when you do not have permission to create a directory.

For example, if hadoop is installed on linux server's licz user, I must use the eclipse Development Program under the licz user on windows.

In this way, we can develop mapreduce programs on eclipse.

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition

Build a Hadoop environment (using virtual machines to build two Ubuntu systems in a Winodws environment)