2.1 Hadoop Eclipse Plugin 配置及安裝

標籤：style   class   blog   c   code   java   

Hadoop Eclipse 開發工具 主要分為

1、根據Hadoop版本產生外掛程式 2、安裝Hadoop Eclipse外掛程式 3、配置Hadoop目錄 4、配置Hadoop串連 5、新一個MapReduce工程 WordCount.javaMapReduce——WordCount問題總結參考：http://blog.sina.com.cn/s/blog_7fcb1aef0100zpux.html正成功輸入出後資訊:

14/05/21 23:06:47 INFO input.FileInputFormat: Total input paths to process : 2
14/05/21 23:06:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/05/21 23:06:47 WARN snappy.LoadSnappy: Snappy native library not loaded
14/05/21 23:06:48 INFO mapred.JobClient: Running job: job_201405220635_0009
14/05/21 23:06:49 INFO mapred.JobClient: map 0% reduce 0%
14/05/21 23:06:59 INFO mapred.JobClient: map 50% reduce 0%
14/05/21 23:07:00 INFO mapred.JobClient: map 100% reduce 0%
14/05/21 23:07:09 INFO mapred.JobClient: map 100% reduce 33%
14/05/21 23:07:11 INFO mapred.JobClient: map 100% reduce 100%
14/05/21 23:07:13 INFO mapred.JobClient: Job complete: job_201405220635_0009
14/05/21 23:07:13 INFO mapred.JobClient: Counters: 29
14/05/21 23:07:13 INFO mapred.JobClient: Job Counters
14/05/21 23:07:13 INFO mapred.JobClient: Launched reduce tasks=1
14/05/21 23:07:13 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=17386
14/05/21 23:07:13 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
14/05/21 23:07:13 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
14/05/21 23:07:13 INFO mapred.JobClient: Launched map tasks=2
14/05/21 23:07:13 INFO mapred.JobClient: Data-local map tasks=2
14/05/21 23:07:13 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=12160
14/05/21 23:07:13 INFO mapred.JobClient: File Output Format Counters
14/05/21 23:07:13 INFO mapred.JobClient: Bytes Written=15
14/05/21 23:07:13 INFO mapred.JobClient: FileSystemCounters
14/05/21 23:07:13 INFO mapred.JobClient: FILE_BYTES_READ=52
14/05/21 23:07:13 INFO mapred.JobClient: HDFS_BYTES_READ=252
14/05/21 23:07:13 INFO mapred.JobClient: FILE_BYTES_WRITTEN=177419
14/05/21 23:07:13 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=15
14/05/21 23:07:13 INFO mapred.JobClient: File Input Format Counters
14/05/21 23:07:13 INFO mapred.JobClient: Bytes Read=22
14/05/21 23:07:13 INFO mapred.JobClient: Map-Reduce Framework
14/05/21 23:07:13 INFO mapred.JobClient: Map output materialized bytes=58
14/05/21 23:07:13 INFO mapred.JobClient: Map input records=2
14/05/21 23:07:13 INFO mapred.JobClient: Reduce shuffle bytes=58
14/05/21 23:07:13 INFO mapred.JobClient: Spilled Records=8
14/05/21 23:07:13 INFO mapred.JobClient: Map output bytes=38
14/05/21 23:07:13 INFO mapred.JobClient: CPU time spent (ms)=5610
14/05/21 23:07:13 INFO mapred.JobClient: Total committed heap usage (bytes)=336338944
14/05/21 23:07:13 INFO mapred.JobClient: Combine input records=4
14/05/21 23:07:13 INFO mapred.JobClient: SPLIT_RAW_BYTES=230
14/05/21 23:07:13 INFO mapred.JobClient: Reduce input records=4
14/05/21 23:07:13 INFO mapred.JobClient: Reduce input groups=2
14/05/21 23:07:13 INFO mapred.JobClient: Combine output records=4
14/05/21 23:07:13 INFO mapred.JobClient: Physical memory (bytes) snapshot=428146688
14/05/21 23:07:13 INFO mapred.JobClient: Reduce output records=2
14/05/21 23:07:13 INFO mapred.JobClient: Virtual memory (bytes) snapshot=2140233728
14/05/21 23:07:13 INFO mapred.JobClient: Map output records=4

 

 

錯誤解決方案：

 

14/05/21 23:14:11 INFO mapred.JobClient: Cleaning up the staging area hdfs://192.168.1.53:9000/app/hadoop/hadoop/tmp/mapred/staging/hadoop/.staging/job_201405220635_0010
14/05/21 23:14:11 ERROR security.UserGroupInformation: PriviledgedActionException as:hadoop cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory newout already exists
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory newout already exists
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:973)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:93)

 

cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory newout already exists

 

說明目錄已經存在了，需要刪除後再執行，命令如下：

hadoop fs -rmr newout