Running Hadoop programs in eclipse

Source: Internet
Author: User
Tags prepare hadoop fs



1, download Hadoop-eclipse-plugin-1.2.1.jar, and copy it to Eclipse/plugins.


2. Open Map-reduce View

In Eclipse, open Window-->open perspetive-->other and select Map/reduce.


3, select Map/reduce Locations tab, create a new location



4. In Project Exploer, you can browse the file system of the site you just defined





5. Prepare test data and upload to HDFs.

Liaoliuqingdemacbook-air:downloads liaoliuqing$ Hadoop Fs-mkdir in

Liaoliuqingdemacbook-air:downloads liaoliuqing$ Hadoop fs-copyfromlocal maxTemp.txt in

Liaoliuqingdemacbook-air:downloads liaoliuqing$ Hadoop Fs-ls in

Found 1 Items

-rw-r--r--1 liaoliuqing supergroup 953 2014-12-14 09:47/user/liaoliuqing/in/maxtemp.txt


The contents of MaxTemp.txt are as follows:

123456798676231190101234567986762311901012345679867623119010123456798676231190101234561+00121534567890356

123456798676231190101234567986762311901012345679867623119010123456798676231190101234562+01122934567890456

123456798676231190201234567986762311901012345679867623119010123456798676231190101234562+02120234567893456

123456798676231190401234567986762311901012345679867623119010123456798676231190101234561+00321234567803456

123456798676231190101234567986762311902012345679867623119010123456798676231190101234561+00429234567903456

123456798676231190501234567986762311902012345679867623119010123456798676231190101234561+01021134568903456

123456798676231190201234567986762311902012345679867623119010123456798676231190101234561+01124234578903456

123456798676231190301234567986762311905012345679867623119010123456798676231190101234561+04121234678903456

123456798676231190301234567986762311905012345679867623119010123456798676231190101234561+00821235678903456


6, Prepare Map-reduce procedure

program Please see http://blog.csdn.net/jediael_lu/article/details/37596469


7. Running the program

Maxtemperature.java-->run As-->run Configuration

Fill in the input and output directories in the arguments and start run.



This is to run the program in HDFs, in fact, you can also run the program in the local file system, this method can be conveniently used for program debugging.

If you fill in the parameters:

/users/liaoliuqing/in/users/liaoliuqing/out

Can.



8, the following is the output content in Eclise console

14/12/14 10:52:05 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable

14/12/14 10:52:05 WARN mapred. Jobclient:use Genericoptionsparser for parsing the arguments. Applications should implement Tool for the same.

14/12/14 10:52:05 WARN mapred.  Jobclient:no job jar file set. User classes May is not found. See jobconf (Class) or Jobconf#setjar (String).

14/12/14 10:52:05 INFO input. Fileinputformat:total input paths to process:1

14/12/14 10:52:05 WARN Snappy. Loadsnappy:snappy Native Library not loaded

14/12/14 10:52:06 INFO mapred. Jobclient:running job:job_local1815770300_0001

14/12/14 10:52:06 INFO mapred. Localjobrunner:waiting for map tasks

14/12/14 10:52:06 INFO mapred. Localjobrunner:starting Task:attempt_local1815770300_0001_m_000000_0

14/12/14 10:52:06 INFO mapred. Task:using Resourcecalculatorplugin:null

14/12/14 10:52:06 INFO mapred. Maptask:processing split:hdfs://localhost:9000/user/liaoliuqing/in/maxtemp.txt:0+953

14/12/14 10:52:06 INFO mapred. MAPTASK:IO.SORT.MB = 100

14/12/14 10:52:06 INFO mapred. Maptask:data buffer = 79691776/99614720

14/12/14 10:52:06 INFO mapred. Maptask:record buffer = 262144/327680

14/12/14 10:52:06 INFO mapred. maptask:starting Flush of map output

14/12/14 10:52:06 INFO mapred. Maptask:finished spill 0

14/12/14 10:52:06 INFO mapred. Task:Task:attempt_local1815770300_0001_m_000000_0 is done. and is in the process of commiting

14/12/14 10:52:06 INFO mapred. Localjobrunner:

14/12/14 10:52:06 INFO mapred. Task:task ' Attempt_local1815770300_0001_m_000000_0 ' done.

14/12/14 10:52:06 INFO mapred. Localjobrunner:finishing Task:attempt_local1815770300_0001_m_000000_0

14/12/14 10:52:06 INFO mapred. Localjobrunner:map Task executor complete.

14/12/14 10:52:06 INFO mapred. Task:using Resourcecalculatorplugin:null

14/12/14 10:52:06 INFO mapred. Localjobrunner:

14/12/14 10:52:06 INFO mapred. Merger:merging 1 sorted Segments

14/12/14 10:52:06 INFO mapred. Merger:down to the last Merge-pass, with 1 segments left of total size:90 bytes

14/12/14 10:52:06 INFO mapred. Localjobrunner:

14/12/14 10:52:06 INFO mapred. Task:Task:attempt_local1815770300_0001_r_000000_0 is done. and is in the process of commiting

14/12/14 10:52:06 INFO mapred. Localjobrunner:

14/12/14 10:52:06 INFO mapred. Task:task Attempt_local1815770300_0001_r_000000_0 is allowed to commit now

14/12/14 10:52:06 INFO output. fileoutputcommitter:saved output of Task ' attempt_local1815770300_0001_r_000000_0 ' to hdfs://localhost:9000/user/ Liaoliuqing/out

14/12/14 10:52:06 INFO mapred. Localjobrunner:reduce > Reduce

14/12/14 10:52:06 INFO mapred. Task:task ' Attempt_local1815770300_0001_r_000000_0 ' done.

14/12/14 10:52:07 INFO mapred. Jobclient:map 100% Reduce 100%

14/12/14 10:52:07 INFO mapred. Jobclient:job complete:job_local1815770300_0001

14/12/14 10:52:07 INFO mapred. Jobclient:counters:19

14/12/14 10:52:07 INFO mapred. Jobclient:file Output Format Counters

14/12/14 10:52:07 INFO mapred. Jobclient:bytes written=43

14/12/14 10:52:07 INFO mapred. Jobclient:file Input Format Counters

14/12/14 10:52:07 INFO mapred. Jobclient:bytes read=953

14/12/14 10:52:07 INFO mapred. Jobclient:filesystemcounters

14/12/14 10:52:07 INFO mapred. jobclient:file_bytes_read=450

14/12/14 10:52:07 INFO mapred. jobclient:hdfs_bytes_read=1906

14/12/14 10:52:07 INFO mapred. jobclient:file_bytes_written=135618

14/12/14 10:52:07 INFO mapred. jobclient:hdfs_bytes_written=43

14/12/14 10:52:07 INFO mapred. Jobclient:map-reduce Framework

14/12/14 10:52:07 INFO mapred. Jobclient:reduce input Groups=5

14/12/14 10:52:07 INFO mapred. Jobclient:map output materialized bytes=94

14/12/14 10:52:07 INFO mapred. Jobclient:combine Output Records=0

14/12/14 10:52:07 INFO mapred. Jobclient:map input Records=9

14/12/14 10:52:07 INFO mapred. Jobclient:reduce Shuffle bytes=0

14/12/14 10:52:07 INFO mapred. Jobclient:reduce Output records=5

14/12/14 10:52:07 INFO mapred. Jobclient:spilled records=16

14/12/14 10:52:07 INFO mapred. Jobclient:map Output bytes=72

14/12/14 10:52:07 INFO mapred. Jobclient:total committed heap usage (bytes) =329252864

14/12/14 10:52:07 INFO mapred. jobclient:split_raw_bytes=118

14/12/14 10:52:07 INFO mapred. Jobclient:map Output records=8

14/12/14 10:52:07 INFO mapred. Jobclient:combine input Records=0


14/12/14 10:52:07 INFO mapred. Jobclient:reduce input Records=8







Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.