1, download Hadoop-eclipse-plugin-1.2.1.jar, and copy it to Eclipse/plugins.
2. Open Map-reduce View
In Eclipse, open Window-->open perspetive-->other and select Map/reduce.
3, select Map/reduce Locations tab, create a new location
4. In Project Exploer, you can browse the file system of the site you just defined
5. Prepare test data and upload to HDFs.
Liaoliuqingdemacbook-air:downloads liaoliuqing$ Hadoop Fs-mkdir in
Liaoliuqingdemacbook-air:downloads liaoliuqing$ Hadoop fs-copyfromlocal maxTemp.txt in
Liaoliuqingdemacbook-air:downloads liaoliuqing$ Hadoop Fs-ls in
Found 1 Items
-rw-r--r--1 liaoliuqing supergroup 953 2014-12-14 09:47/user/liaoliuqing/in/maxtemp.txt
The contents of MaxTemp.txt are as follows:
123456798676231190101234567986762311901012345679867623119010123456798676231190101234561+00121534567890356
123456798676231190101234567986762311901012345679867623119010123456798676231190101234562+01122934567890456
123456798676231190201234567986762311901012345679867623119010123456798676231190101234562+02120234567893456
123456798676231190401234567986762311901012345679867623119010123456798676231190101234561+00321234567803456
123456798676231190101234567986762311902012345679867623119010123456798676231190101234561+00429234567903456
123456798676231190501234567986762311902012345679867623119010123456798676231190101234561+01021134568903456
123456798676231190201234567986762311902012345679867623119010123456798676231190101234561+01124234578903456
123456798676231190301234567986762311905012345679867623119010123456798676231190101234561+04121234678903456
123456798676231190301234567986762311905012345679867623119010123456798676231190101234561+00821235678903456
6, Prepare Map-reduce procedure
program Please see http://blog.csdn.net/jediael_lu/article/details/37596469
7. Running the program
Maxtemperature.java-->run As-->run Configuration
Fill in the input and output directories in the arguments and start run.
This is to run the program in HDFs, in fact, you can also run the program in the local file system, this method can be conveniently used for program debugging.
If you fill in the parameters:
/users/liaoliuqing/in/users/liaoliuqing/out
Can.
8, the following is the output content in Eclise console
14/12/14 10:52:05 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable
14/12/14 10:52:05 WARN mapred. Jobclient:use Genericoptionsparser for parsing the arguments. Applications should implement Tool for the same.
14/12/14 10:52:05 WARN mapred. Jobclient:no job jar file set. User classes May is not found. See jobconf (Class) or Jobconf#setjar (String).
14/12/14 10:52:05 INFO input. Fileinputformat:total input paths to process:1
14/12/14 10:52:05 WARN Snappy. Loadsnappy:snappy Native Library not loaded
14/12/14 10:52:06 INFO mapred. Jobclient:running job:job_local1815770300_0001
14/12/14 10:52:06 INFO mapred. Localjobrunner:waiting for map tasks
14/12/14 10:52:06 INFO mapred. Localjobrunner:starting Task:attempt_local1815770300_0001_m_000000_0
14/12/14 10:52:06 INFO mapred. Task:using Resourcecalculatorplugin:null
14/12/14 10:52:06 INFO mapred. Maptask:processing split:hdfs://localhost:9000/user/liaoliuqing/in/maxtemp.txt:0+953
14/12/14 10:52:06 INFO mapred. MAPTASK:IO.SORT.MB = 100
14/12/14 10:52:06 INFO mapred. Maptask:data buffer = 79691776/99614720
14/12/14 10:52:06 INFO mapred. Maptask:record buffer = 262144/327680
14/12/14 10:52:06 INFO mapred. maptask:starting Flush of map output
14/12/14 10:52:06 INFO mapred. Maptask:finished spill 0
14/12/14 10:52:06 INFO mapred. Task:Task:attempt_local1815770300_0001_m_000000_0 is done. and is in the process of commiting
14/12/14 10:52:06 INFO mapred. Localjobrunner:
14/12/14 10:52:06 INFO mapred. Task:task ' Attempt_local1815770300_0001_m_000000_0 ' done.
14/12/14 10:52:06 INFO mapred. Localjobrunner:finishing Task:attempt_local1815770300_0001_m_000000_0
14/12/14 10:52:06 INFO mapred. Localjobrunner:map Task executor complete.
14/12/14 10:52:06 INFO mapred. Task:using Resourcecalculatorplugin:null
14/12/14 10:52:06 INFO mapred. Localjobrunner:
14/12/14 10:52:06 INFO mapred. Merger:merging 1 sorted Segments
14/12/14 10:52:06 INFO mapred. Merger:down to the last Merge-pass, with 1 segments left of total size:90 bytes
14/12/14 10:52:06 INFO mapred. Localjobrunner:
14/12/14 10:52:06 INFO mapred. Task:Task:attempt_local1815770300_0001_r_000000_0 is done. and is in the process of commiting
14/12/14 10:52:06 INFO mapred. Localjobrunner:
14/12/14 10:52:06 INFO mapred. Task:task Attempt_local1815770300_0001_r_000000_0 is allowed to commit now
14/12/14 10:52:06 INFO output. fileoutputcommitter:saved output of Task ' attempt_local1815770300_0001_r_000000_0 ' to hdfs://localhost:9000/user/ Liaoliuqing/out
14/12/14 10:52:06 INFO mapred. Localjobrunner:reduce > Reduce
14/12/14 10:52:06 INFO mapred. Task:task ' Attempt_local1815770300_0001_r_000000_0 ' done.
14/12/14 10:52:07 INFO mapred. Jobclient:map 100% Reduce 100%
14/12/14 10:52:07 INFO mapred. Jobclient:job complete:job_local1815770300_0001
14/12/14 10:52:07 INFO mapred. Jobclient:counters:19
14/12/14 10:52:07 INFO mapred. Jobclient:file Output Format Counters
14/12/14 10:52:07 INFO mapred. Jobclient:bytes written=43
14/12/14 10:52:07 INFO mapred. Jobclient:file Input Format Counters
14/12/14 10:52:07 INFO mapred. Jobclient:bytes read=953
14/12/14 10:52:07 INFO mapred. Jobclient:filesystemcounters
14/12/14 10:52:07 INFO mapred. jobclient:file_bytes_read=450
14/12/14 10:52:07 INFO mapred. jobclient:hdfs_bytes_read=1906
14/12/14 10:52:07 INFO mapred. jobclient:file_bytes_written=135618
14/12/14 10:52:07 INFO mapred. jobclient:hdfs_bytes_written=43
14/12/14 10:52:07 INFO mapred. Jobclient:map-reduce Framework
14/12/14 10:52:07 INFO mapred. Jobclient:reduce input Groups=5
14/12/14 10:52:07 INFO mapred. Jobclient:map output materialized bytes=94
14/12/14 10:52:07 INFO mapred. Jobclient:combine Output Records=0
14/12/14 10:52:07 INFO mapred. Jobclient:map input Records=9
14/12/14 10:52:07 INFO mapred. Jobclient:reduce Shuffle bytes=0
14/12/14 10:52:07 INFO mapred. Jobclient:reduce Output records=5
14/12/14 10:52:07 INFO mapred. Jobclient:spilled records=16
14/12/14 10:52:07 INFO mapred. Jobclient:map Output bytes=72
14/12/14 10:52:07 INFO mapred. Jobclient:total committed heap usage (bytes) =329252864
14/12/14 10:52:07 INFO mapred. jobclient:split_raw_bytes=118
14/12/14 10:52:07 INFO mapred. Jobclient:map Output records=8
14/12/14 10:52:07 INFO mapred. Jobclient:combine input Records=0
14/12/14 10:52:07 INFO mapred. Jobclient:reduce input Records=8