2.1 Hadoop Eclipse Plugin Configuration and installation

Hadoop Eclipse development tools are mainly divided into

1. Build Plugin 2 According to Hadoop version, install Hadoop Eclipse plug-in 3, configure Hadoop directory 4, configure Hadoop connection 5, a new MapReduce project Wordcount.javaSummary of Mapreduce--wordcount problemsReference: Http://blog.sina.com.cn/s/blog_7fcb1aef0100zpux.htmlThe following information is being successfully entered:

14/05/21 23:06:47 INFO input. Fileinputformat:total input paths to Process:2
14/05/21 23:06:47 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable
14/05/21 23:06:47 WARN Snappy. Loadsnappy:snappy Native library not loaded
14/05/21 23:06:48 INFO mapred. Jobclient:running job:job_201405220635_0009
14/05/21 23:06:49 INFO mapred. Jobclient:map 0% reduce 0%
14/05/21 23:06:59 INFO mapred. Jobclient:map 50% reduce 0%
14/05/21 23:07:00 INFO mapred. Jobclient:map 100% reduce 0%
14/05/21 23:07:09 INFO mapred. Jobclient:map 100% reduce 33%
14/05/21 23:07:11 INFO mapred. Jobclient:map 100% reduce 100%
14/05/21 23:07:13 INFO mapred. Jobclient:job complete:job_201405220635_0009
14/05/21 23:07:13 INFO mapred. Jobclient:counters:29
14/05/21 23:07:13 INFO mapred. Jobclient:job Counters
14/05/21 23:07:13 INFO mapred. jobclient:launched Reduce Tasks=1
14/05/21 23:07:13 INFO mapred. Jobclient:slots_millis_maps=17386
14/05/21 23:07:13 INFO mapred. Jobclient:total time spent by all reduces waiting after reserving slots (ms) =0
14/05/21 23:07:13 INFO mapred. Jobclient:total time spent by all maps waiting after reserving slots (ms) =0
14/05/21 23:07:13 INFO mapred. jobclient:launched Map tasks=2
14/05/21 23:07:13 INFO mapred. Jobclient:data-local Map tasks=2
14/05/21 23:07:13 INFO mapred. jobclient:slots_millis_reduces=12160
14/05/21 23:07:13 INFO mapred. Jobclient:file Output Format Counters
14/05/21 23:07:13 INFO mapred. Jobclient:bytes written=15
14/05/21 23:07:13 INFO mapred. Jobclient:filesystemcounters
14/05/21 23:07:13 INFO mapred. jobclient:file_bytes_read=52
14/05/21 23:07:13 INFO mapred. jobclient:hdfs_bytes_read=252
14/05/21 23:07:13 INFO mapred. jobclient:file_bytes_written=177419
14/05/21 23:07:13 INFO mapred. Jobclient:hdfs_bytes_written=15
14/05/21 23:07:13 INFO mapred. Jobclient:file Input Format Counters
14/05/21 23:07:13 INFO mapred. Jobclient:bytes read=22
14/05/21 23:07:13 INFO mapred. Jobclient:map-reduce Framework
14/05/21 23:07:13 INFO mapred. Jobclient:map output materialized bytes=58
14/05/21 23:07:13 INFO mapred. Jobclient:map input records=2
14/05/21 23:07:13 INFO mapred. Jobclient:reduce Shuffle bytes=58
14/05/21 23:07:13 INFO mapred. jobclient:spilled records=8
14/05/21 23:07:13 INFO mapred. Jobclient:map Output bytes=38
14/05/21 23:07:13 INFO mapred. Jobclient:cpu Time Spent (ms) =5610
14/05/21 23:07:13 INFO mapred. Jobclient:total committed heap usage (bytes) =336338944
14/05/21 23:07:13 INFO mapred. Jobclient:combine input records=4
14/05/21 23:07:13 INFO mapred. jobclient:split_raw_bytes=230
14/05/21 23:07:13 INFO mapred. Jobclient:reduce input records=4
14/05/21 23:07:13 INFO mapred. Jobclient:reduce input groups=2
14/05/21 23:07:13 INFO mapred. Jobclient:combine Output records=4
14/05/21 23:07:13 INFO mapred. Jobclient:physical memory (bytes) snapshot=428146688
14/05/21 23:07:13 INFO mapred. Jobclient:reduce Output records=2
14/05/21 23:07:13 INFO mapred. Jobclient:virtual memory (bytes) snapshot=2140233728
14/05/21 23:07:13 INFO mapred. Jobclient:map Output records=4

Error Resolution Method:

14/05/21 23:14:11 INFO mapred. Jobclient:cleaning up the staging area hdfs://192.168.1.53:9000/app/hadoop/hadoop/tmp/mapred/staging/hadoop/. staging/job_201405220635_0010
14/05/21 23:14:11 ERROR security. Usergroupinformation:priviledgedactionexception As:hadoop Cause:o Rg.apache.hadoop.mapred.FileAlreadyExistsException:Output directory newout already exists
Exception in thread "main" Org.apache.hadoop.mapred.FileAlreadyExistsException:Output directory Newout already Exists
At org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs (fileoutputformat.java:137)
At Org.apache.hadoop.mapred.jobclient$2.run (jobclient.java:973)
At Org.apache.hadoop.mapred.jobclient$2.run (jobclient.java:936)
At java.security.AccessController.doPrivileged (Native Method)
At javax.security.auth.Subject.doAs (subject.java:396)
At org.apache.hadoop.security.UserGroupInformation.doAs (usergroupinformation.java:1190)
At org.apache.hadoop.mapred.JobClient.submitJobInternal (jobclient.java:936)
At Org.apache.hadoop.mapreduce.Job.submit (job.java:550)
At org.apache.hadoop.mapreduce.Job.waitForCompletion (job.java:580)
At Org.apache.hadoop.examples.WordCount.main (wordcount.java:93)

Cause:org.apache.hadoop.mapred.FileAlreadyExistsException:Output directory Newout already exists

The description directory already exists and needs to be deleted before executing, the command is as follows:

Hadoop FS-RMR Newout