1, download Hadoop-eclipse-plugin-1.2.1.jar, and copy it to Eclipse/plugins.
2. Open Map-reduce View
In Eclipse, open Window-->open perspetive-->other and select Map/reduce.
3, select Map/reduce Locations tab, create a new location
4. In Project Exploer, you can browse the file system of the site you just defined
5. Prepare test data and upload to HDFs.
Liaoliuqingdemacbook-air:downloads liaoliuqing$ Hadoop Fs-mkdir in
Liaoliuqingdemacbook-air:downloads liaoliuqing$ Hadoop fs-copyfromlocal maxTemp.txt in
Liaoliuqingdemacbook-air:downloads liaoliuqing$ Hadoop Fs-ls in
Found 1 Items
-rw-r--r--1 liaoliuqing supergroup 953 2014-12-14 09:47/user/liaoliuqing/in/maxtemp.txt
The contents of MaxTemp.txt are as follows:
123456798676231190101234567986762311901012345679867623119010123456798676231190101234561+00121534567890356
123456798676231190101234567986762311901012345679867623119010123456798676231190101234562+01122934567890456
123456798676231190201234567986762311901012345679867623119010123456798676231190101234562+02120234567893456
123456798676231190401234567986762311901012345679867623119010123456798676231190101234561+00321234567803456
123456798676231190101234567986762311902012345679867623119010123456798676231190101234561+00429234567903456
123456798676231190501234567986762311902012345679867623119010123456798676231190101234561+01021134568903456
123456798676231190201234567986762311902012345679867623119010123456798676231190101234561+01124234578903456
123456798676231190301234567986762311905012345679867623119010123456798676231190101234561+04121234678903456
123456798676231190301234567986762311905012345679867623119010123456798676231190101234561+00821235678903456
6, Prepare Map-reduce procedure
program please see http://blog.csdn.net/jediael_lu/article/details/37596469
7. Running the program
Maxtemperature.java-->run As-->run Configuration
Fill in the input and output documents in the arguments and start run.
8, the following is the output content in Eclise console
14/12/14 10:52:05 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable
14/12/14 10:52:05 WARN mapred. Jobclient:use Genericoptionsparser for parsing the arguments. Applications should implement Tool for the same.
14/12/14 10:52:05 WARN mapred. Jobclient:no job jar file set. User classes May is not found. See jobconf (Class) or Jobconf#setjar (String).
14/12/14 10:52:05 INFO input. Fileinputformat:total input paths to process:1
14/12/14 10:52:05 WARN Snappy. Loadsnappy:snappy Native Library not loaded
14/12/14 10:52:06 INFO mapred. Jobclient:running job:job_local1815770300_0001
14/12/14 10:52:06 INFO mapred. Localjobrunner:waiting for map tasks
14/12/14 10:52:06 INFO mapred. Localjobrunner:starting Task:attempt_local1815770300_0001_m_000000_0
14/12/14 10:52:06 INFO mapred. Task:using Resourcecalculatorplugin:null
14/12/14 10:52:06 INFO mapred. Maptask:processing split:hdfs://localhost:9000/user/liaoliuqing/in/maxtemp.txt:0+953
14/12/14 10:52:06 INFO mapred. MAPTASK:IO.SORT.MB = 100
14/12/14 10:52:06 INFO mapred. Maptask:data buffer = 79691776/99614720
14/12/14 10:52:06 INFO mapred. Maptask:record buffer = 262144/327680
14/12/14 10:52:06 INFO mapred. maptask:starting Flush of map output
14/12/14 10:52:06 INFO mapred. Maptask:finished spill 0
14/12/14 10:52:06 INFO mapred. Task:Task:attempt_local1815770300_0001_m_000000_0 is done. and is in the process of commiting
14/12/14 10:52:06 INFO mapred. Localjobrunner:
14/12/14 10:52:06 INFO mapred. Task:task ' Attempt_local1815770300_0001_m_000000_0 ' done.
14/12/14 10:52:06 INFO mapred. Localjobrunner:finishing Task:attempt_local1815770300_0001_m_000000_0
14/12/14 10:52:06 INFO mapred. Localjobrunner:map Task executor complete.
14/12/14 10:52:06 INFO mapred. Task:using Resourcecalculatorplugin:null
14/12/14 10:52:06 INFO mapred. Localjobrunner:
14/12/14 10:52:06 INFO mapred. Merger:merging 1 sorted Segments
14/12/14 10:52:06 INFO mapred. Merger:down to the last Merge-pass, with 1 segments left of total size:90 bytes
14/12/14 10:52:06 INFO mapred. Localjobrunner:
14/12/14 10:52:06 INFO mapred. Task:Task:attempt_local1815770300_0001_r_000000_0 is done. and is in the process of commiting
14/12/14 10:52:06 INFO mapred. Localjobrunner:
14/12/14 10:52:06 INFO mapred. Task:task Attempt_local1815770300_0001_r_000000_0 is allowed to commit now
14/12/14 10:52:06 INFO output. fileoutputcommitter:saved output of Task ' attempt_local1815770300_0001_r_000000_0 ' to hdfs://localhost:9000/user/ Liaoliuqing/out
14/12/14 10:52:06 INFO mapred. Localjobrunner:reduce > Reduce
14/12/14 10:52:06 INFO mapred. Task:task ' Attempt_local1815770300_0001_r_000000_0 ' done.
14/12/14 10:52:07 INFO mapred. Jobclient:map 100% Reduce 100%
14/12/14 10:52:07 INFO mapred. Jobclient:job complete:job_local1815770300_0001
14/12/14 10:52:07 INFO mapred. Jobclient:counters:19
14/12/14 10:52:07 INFO mapred. Jobclient:file Output Format Counters
14/12/14 10:52:07 INFO mapred. Jobclient:bytes written=43
14/12/14 10:52:07 INFO mapred. Jobclient:file Input Format Counters
14/12/14 10:52:07 INFO mapred. Jobclient:bytes read=953
14/12/14 10:52:07 INFO mapred. Jobclient:filesystemcounters
14/12/14 10:52:07 INFO mapred. jobclient:file_bytes_read=450
14/12/14 10:52:07 INFO mapred. jobclient:hdfs_bytes_read=1906
14/12/14 10:52:07 INFO mapred. jobclient:file_bytes_written=135618
14/12/14 10:52:07 INFO mapred. jobclient:hdfs_bytes_written=43
14/12/14 10:52:07 INFO mapred. Jobclient:map-reduce Framework
14/12/14 10:52:07 INFO mapred. Jobclient:reduce input Groups=5
14/12/14 10:52:07 INFO mapred. Jobclient:map output materialized bytes=94
14/12/14 10:52:07 INFO mapred. Jobclient:combine Output Records=0
14/12/14 10:52:07 INFO mapred. Jobclient:map input Records=9
14/12/14 10:52:07 INFO mapred. Jobclient:reduce Shuffle bytes=0
14/12/14 10:52:07 INFO mapred. Jobclient:reduce Output records=5
14/12/14 10:52:07 INFO mapred. Jobclient:spilled records=16
14/12/14 10:52:07 INFO mapred. Jobclient:map Output bytes=72
14/12/14 10:52:07 INFO mapred. Jobclient:total committed heap usage (bytes) =329252864
14/12/14 10:52:07 INFO mapred. jobclient:split_raw_bytes=118
14/12/14 10:52:07 INFO mapred. Jobclient:map Output records=8
14/12/14 10:52:07 INFO mapred. Jobclient:combine input Records=0
14/12/14 10:52:07 INFO mapred. Jobclient:reduce input Records=8
Running Hadoop programs in eclipse