Implementing Hadoop Wordcount.jar under Linux

Linux executes Hadoop WordCount

Ubuntu Terminal Access shortcut key: Ctrl + ALT +t

Hadoop startup command:start-all.sh

The normal execution effect is as follows:

[Email protected]:~$ start-all.sh

Warning: $HADOOP _home is deprecated.

Starting Namenode, logging to/home/hadoop/hadoop-1.1.2/libexec/. /logs/hadoop-hadoop-namenode-hadoop.main.out

Hadoop. Main:starting Datanode, logging to/home/hadoop/hadoop-1.1.2/libexec/. /logs/hadoop-hadoop-datanode-hadoop.main.out

Hadoop. Main:starting secondarynamenode,logging to/home/hadoop/hadoop-1.1.2/libexec/. /logs/hadoop-hadoop-secondarynamenode-hadoop.main.out

Starting Jobtracker, logging to/home/hadoop/hadoop-1.1.2/libexec/. /logs/hadoop-hadoop-jobtracker-hadoop.main.out

Hadoop. Main:starting Tasktracker, loggingto/home/hadoop/hadoop-1.1.2/libexec/. /logs/hadoop-hadoop-tasktracker-hadoop.main.out

JPS command to view the started Hadoop service

[Email protected]:~$ JPS

3615 Jps

2699 NameNode

3461 Tasktracker

2922 DataNode

3137 Secondarynamenode

3231 Jobtracker

Create a folder locally

[Email protected]:~$ mkdir ~/file

Create two txt files in a file

[Email protected]:~$ CD file

[Email protected]:~/file$ echo "Hello World" > File1.txt

[Email protected]:~/file$ echo "Hello Hadoop" > File2.txt

[Email protected]:~/file$ ls

File1.txt File2.txt

[Email protected]:~/file$

Create an input folder on HDFs

[Email protected]:~/file$ Hadoop fs-mkdir input

View the input folder path created

[Email protected]:~$ Hadoop fs-ls

Warning: $HADOOP _home is deprecated.

Found 5 Items

-rw-r--r--3 Administrator supergroup 6296230 2014-09-03 10:38/user/hadoop/cloud.txt

Drwxr-xr-x-hadoop supergroup 0 2014-09-02 16:31/USER/HADOOP/HADI_CURBM

Drwxr-xr-x-hadoop supergroup 0 2014-09-04 09:59 /user/hadoop/input

Drwxr-xr-x-hadoop supergroup 0 2014-09-02 16:31/user/hadoop/pegasus

[Email protected]:~$

You can see that the directory was created to the/user/hadoop/input directory

Uploading local file files to the input directory

[Email protected]:~ $hadoop FS put ~/file/*.txt /user/hadoop/input

Find the Examples.jar package in the Hadoop directory

[Email protected]:~$ CD hadoop-1.1.2

[Email protected]:~/hadoop-1.1.2$ ls

Bin Docs Hadoop-test-1.1.2.jar LICENSE.txt src

Build.xml Hadoop-ant-1.1.2.jar Hadoop-tools-1.1.2.jar Logs WebApps

C + + Hadoop-client-1.1.2.jar Ivy NOTICE.txt Wordcount.jar

CHANGES.txt Hadoop-core-1.1.2.jar Ivy.xml README.txt

Conf Hadoop-examples-1.1.2.jar Lib Sbin

Contrib Hadoop-minicluster-1.1.2.jar libexec Share

[Email protected]:~/hadoop-1.1.2$

Execute JAR Program code statistics wordcount of files in input directory

[Email protected]:~$ Hadoop Jar/home/hadoop/hadoop-1.1.2/hadoop-examples-1.1.2.jar wordcount/user/hadoop/input Output

Warning: $HADOOP _home is deprecated.

14/09/04 10:10:44 INFOinput.FileInputFormat:Total input paths to process:0

14/09/04 10:10:45 INFO mapred. Jobclient:running job:job_201409040943_0001

14/09/04 10:10:46 INFOmapred.JobClient:map 0% reduce 0%

14/09/04 10:10:54 INFOmapred.JobClient:map 0% reduce 100%

14/09/04 10:10:55 INFO mapred. Jobclient:job complete:job_201409040943_0001

14/09/04 10:10:55 INFO mapred. Jobclient:counters:18

14/09/04 10:10:55 INFOmapred.JobClient:Job Counters

14/09/04 10:10:55 INFOmapred.JobClient:Launched Reducetasks=1

14/09/04 10:10:55 infomapred.jobclient:slots_millis_maps=4087

14/09/04 10:10:55 INFOmapred.JobClient:Total time spent byall reduces waiting after reserving slots (ms) =0

14/09/04 10:10:55 INFOmapred.JobClient:Total Time spent byall maps waiting after reserving slots (ms) =0

14/09/04 10:10:55 INFO mapred. jobclient:slots_millis_reduces=4068

14/09/04 10:10:55 INFOmapred.JobClient:File Output formatcounters

14/09/04 10:10:55 INFOmapred.JobClient:Bytes written=0

14/09/04 10:10:55 INFOmapred.JobClient:FileSystemCounters

14/09/04 10:10:55 INFO mapred. jobclient:file_bytes_written=55309

14/09/04 10:10:55 infomapred.jobclient:map-reduce Framework

14/09/04 10:10:55 INFOmapred.JobClient:Reduce inputgroups=0

14/09/04 10:10:55 INFOmapred.JobClient:Combine outputrecords=0

14/09/04 10:10:55 INFOmapred.JobClient:Reduce shufflebytes=0

14/09/04 10:10:55 INFOmapred.JobClient:Physical Memory (bytes) snapshot=35037184

14/09/04 10:10:55 INFOmapred.JobClient:Reduce outputrecords=0

14/09/04 10:10:55 INFO mapred. Jobclient:spilled records=0

14/09/04 10:10:55 INFOmapred.JobClient:CPU Time Spent (ms) =120

14/09/04 10:10:55 INFOmapred.JobClient:Total committedheap usage (bytes) =15925248

14/09/04 10:10:55 INFOmapred.JobClient:Virtual Memory (bytes) snapshot=377499648

14/09/04 10:10:55 INFOmapred.JobClient:Combine inputrecords=0

14/09/04 10:10:55 INFOmapred.JobClient:Reduce inputrecords=0

[Email protected]:~$

Show results

[Email protected]:~$ hadoop fs-ls output

Warning: $HADOOP _home is deprecated.

Found 3 Items

-rw-r--r--1 Hadoop supergroup 02014-09-04 10:10/user/hadoop/output/_success

Drwxr-xr-x-Hadoop supergroup 02014-09-04 10:10/user/hadoop/output/_logs

-rw-r--r--1 Hadoop supergroup 02014-09-04 10:10 /user/hadoop/output/part-r-00000

[Email protected]:~$

View execution Results

[Email protected]:~$ Hadoop fs-cat output/part-r-00000

Hadoop 1

Hello 2

World 1

Implementing Hadoop Wordcount.jar under Linux