mvn+eclipse構建hadoop項目並運行(超簡單hadoop開發入門指南)

標籤：hadoop   開發環境   

本文詳述如何在windows開發環境下通過mvn+eclipse構建hadoop項目並運行

必備環境

• windows7作業系統
• eclipse-4.4.2
• mvn-3.0.3及用mvn產生項目架構(參閱http://blog.csdn.net/tang9140/article/details/39157439)
• hadoop-2.5.2（直接上hadoop官網http://hadoop.apache.org/下載hadoop-2.5.2.tar.gz並解壓到某個目錄）

windows7下環境配置

1、本地hadoop環境配置
添加環境變數HADOOP_HOME=E:\doc_api\ebook\hadoop-2.5.2
追加環境變數path內容：%HADOOP_HOME%\bin

2、bin下增加hadoop.dll,winutils.exe檔案
從https://github.com/srccodes/hadoop-common-2.2.0-bin或從……下載hadoop.dll,winutils.exe，放置到${HADOOP_HOME}\bin目錄下

構建hadoop項目

下面以經典的WordCount為例，構建我們第一個hadoop項目。

• 引包

pom檔案中加入依賴包

<dependency>    <groupId>org.apache.hadoop</groupId>    <artifactId>hadoop-mapreduce-client-core</artifactId>    <version>2.5.2</version></dependency><dependency>    <groupId>org.apache.hadoop</groupId>    <artifactId>hadoop-common</artifactId>    <version>2.5.2</version></dependency><dependency>    <groupId>org.apache.hadoop</groupId>    <artifactId>hadoop-hdfs</artifactId>    <version>2.5.2</version></dependency><dependency>    <groupId>org.apache.hadoop</groupId>    <artifactId>hadoop-mapreduce-client-common</artifactId>    <version>2.5.2</version></dependency>

• 編寫WordCount類如下

import java.io.IOException;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.conf.Configured;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;import org.apache.hadoop.util.Tool;import org.apache.hadoop.util.ToolRunner;/** * @version 1.0 * @author tangqian */public class WordCount extends Configured implements Tool {    public static void main(String[] args) throws Exception {        int result = ToolRunner.run(new Configuration(),new WordCount(), args);        System.exit(result);    }    @Override    public int run(String[] args) throws Exception {        Path inputPath, outputPath;        if(args.length == 2){            inputPath = new Path(args[0]);            outputPath = new Path(args[1]);        }else{            System.out.println("usage <input> <output>");            return 1;        }        Configuration conf = getConf();        Job job = Job.getInstance(conf, "word count");        job.setJarByClass(WordCount.class);        job.setMapperClass(WordCountMapper.class);        job.setReducerClass(WordCountReducer.class);        job.setInputFormatClass(TextInputFormat.class);        job.setOutputFormatClass(TextOutputFormat.class);        job.setOutputKeyClass(Text.class);        job.setOutputValueClass(IntWritable.class);        FileInputFormat.addInputPath(job, inputPath);        FileOutputFormat.setOutputPath(job, outputPath);        return job.waitForCompletion(true) ? 0 : 1;    }    public static class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {        private final static IntWritable one = new IntWritable(1);        private Text word = new Text();        @Override        public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {            StringTokenizer itr = new StringTokenizer(value.toString());            while (itr.hasMoreTokens()) {                word.set(itr.nextToken());                context.write(word, one);            }        }    }    public static class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {        private IntWritable result = new IntWritable();        @Override        public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {            int sum = 0;            for (IntWritable value : values) {                sum += value.get();            }            result.set(sum);            context.write(key, result);        }    }}

然後在該類上右鍵Run As->Run Configurations->Arguments標籤的Program arguments中指定輸入路徑和輸出路徑如下：

file:///e:/word.txt file:///e:/hadoop/result2

點Run即可運行該類，此時可在Console看到輸出資訊。等完成後，可到e:/hadoop/result2看到結果檔案part-r-00000內容如下

is  1test    2this    1two 1

說明：由於是在本地hadoop單機模式下運行，故採用本地檔案系統（以file://開頭指定輸入輸出路徑）。

附
hadoop-2.5.2叢集安裝指南(參閱http://blog.csdn.net/tang9140/article/details/42869531)

如何修改Windows7下的hosts檔案？
hosts檔案一般在C:\Windows\System32\drivers\etc目錄下，在windows7下如果不是管理員身份登入，可能無許可權修改，此時可右鍵hosts檔案->屬性->安全->編輯，選擇當前登入使用者，開放修改許可權即可，具體操作如。

mvn+eclipse構建hadoop項目並運行(超簡單hadoop開發入門指南)