Run Hadoop WordCount. jar in Linux.

Run Hadoop WordCount. jar in Linux.

Run Hadoop WordCount in Linux

Enter the shortcut key of Ubuntu terminal: ctrl + Alt + t

Hadoop launch command: start-all.sh

 

The normal execution results are as follows:

Hadoop @ HADOOP :~ $ Start-all.sh

Warning: $ HADOOP_HOME is deprecated.

 

Starting namenode, logging to/home/hadoop/hadoop-1.1.2/libexec/../logs/hadoop-hadoop-namenode-HADOOP.MAIN.out

HADOOP. MAIN: starting datanode, logging to/home/hadoop/hadoop-1.1.2/libexec/../logs/hadoop-hadoop-datanode-HADOOP.MAIN.out

HADOOP. MAIN: starting secondarynamenode, logging to/home/hadoop/hadoop-1.1.2/libexec/../logs/hadoop-hadoop-secondarynamenode-HADOOP.MAIN.out

Starting jobtracker, logging to/home/hadoop/hadoop-1.1.2/libexec/../logs/hadoop-hadoop-jobtracker-HADOOP.MAIN.out

HADOOP. MAIN: starting tasktracker, loggingto/home/hadoop/hadoop-1.1.2/libexec/../logs/hadoop-hadoop-tasktracker-HADOOP.MAIN.out

 

Run the jps command to view the started hadoop service.

Hadoop @ HADOOP :~ $ Jps

Jps 3615

2699 NameNode

3461 TaskTracker

2922 DataNode

3137 SecondaryNameNode

3231 JobTracker

 

Create a local folder

Hadoop @ HADOOP :~ $ Mkdir ~ /File

 

Create two txt files in the file

Hadoop @ HADOOP :~ $ Cd file

Hadoop @ HADOOP :~ /File $ echo "Hello World"> file1.txt

Hadoop @ HADOOP :~ /File $ echo "Hello Hadoop"> file2.txt

Hadoop @ HADOOP :~ /File $ ls

File1.txt file2.txt

Hadoop @ HADOOP :~ /File $

 

Create an Input Folder on HDFS

Hadoop @ HADOOP :~ /File $ hadoop fs-mkdir input

View the path of the created input Folder

Hadoop @ HADOOP :~ $ Hadoop fs-ls

Warning: $ HADOOP_HOME is deprecated.

 

Found 5 items

-Rw-r -- 3 Administrator supergroup 6296230 2014-09-03 10: 38/user/hadoop/cloud.txt

Drwxr-xr-x-hadoop supergroup 0 2014-09-02 16:31/user/hadoop/hadi_curbm

Drwxr-xr-x-hadoop supergroup 0 2014-09-04 09:59/user/hadoop/input

Drwxr-xr-x-hadoop supergroup 0 2014-09-02 16:31/user/hadoop/pegasus

Hadoop @ HADOOP :~ $

 

The directory is created to the/user/hadoop/input directory.

 

Upload a local file to the input directory

Hadoop @ HADOOP :~ $ Hadoop fs put ~ /File/*. txt/user/hadoop/input

 

Find the examples. jar package in the hadoop directory.

Hadoop @ HADOOP :~ $ Hadoop-1.1.2 cd

Hadoop @ HADOOP :~ Hadoop-1.1.2 $ ls

Bin docs hadoop-test-1.1.2.jar LICENSE.txt src

Build. xml hadoop-ant-1.1.2.jar hadoop-tools-1.1.2.jar logs webapps

C + + hadoop-client-1.1.2.jar ivy NOTICE.txt wordcount. jar

CHANGES.txt hadoop-core-1.1.2.jar ivy. xml README.txt

Conf hadoop-examples-1.1.2.jar lib sbin

Contrib hadoop-minicluster-1.1.2.jar libexec share

Hadoop @ HADOOP :~ Hadoop-1.1.2 $

 

Execute the jar program code to count the Wordcount of the files in the input directory

Hadoop @ HADOOP :~ $ Hadoop jar/home/hadoop/hadoop-1.1.2/hadoop-examples-1.1.2.jar wordcount/user/hadoop/input output

Warning: $ HADOOP_HOME is deprecated.

 

14/09/04 10:10:44 INFOinput. FileInputFormat: Total input paths to process: 0

14/09/04 10:10:45 INFO mapred. JobClient: Running job: job_201409040943_0001

14/09/04 10:10:46 INFOmapred. JobClient: map 0% reduce 0%

14/09/04 10:10:54 INFOmapred. JobClient: map 0% reduce 100%

14/09/04 10:10:55 INFO mapred. JobClient: Job complete: job_201409040943_0001

14/09/04 10:10:55 INFO mapred. JobClient: Counters: 18

14/09/04 10:10:55 INFOmapred. JobClient: Job Counters

14/09/04 10:10:55 INFOmapred. JobClient: Launched reducetasks = 1

14/09/04 10:10:55 INFOmapred. JobClient: SLOTS_MILLIS_MAPS = 4087

14/09/04 10:10:55 INFOmapred. JobClient: Total time spent byall CES waiting after reserving slots (MS) = 0

14/09/04 10:10:55 INFOmapred. JobClient: Total time spent byall maps waiting after reserving slots (MS) = 0

14/09/04 10:10:55 INFO mapred. JobClient: SLOTS_MILLIS_REDUCES = 4068

14/09/04 10:10:55 INFOmapred. JobClient: File Output FormatCounters

14/09/04 10:10:55 INFOmapred. JobClient: Bytes Written = 0

14/09/04 10:10:55 INFOmapred. JobClient: FileSystemCounters

14/09/04 10:10:55 INFO mapred. JobClient: FILE_BYTES_WRITTEN = 55309

14/09/04 10:10:55 INFOmapred. JobClient: Map-Reduce Framework

14/09/04 10:10:55 INFOmapred. JobClient: Reduce inputgroups = 0

14/09/04 10:10:55 INFOmapred. JobClient: Combine outputrecords = 0

14/09/04 10:10:55 INFOmapred. JobClient: Reduce shufflebytes = 0

14/09/04 10:10:55 INFOmapred. JobClient: Physical memory (bytes) snapshot = 35037184

14/09/04 10:10:55 INFOmapred. JobClient: Reduce outputrecords = 0

14/09/04 10:10:55 INFO mapred. JobClient: Spilled Records = 0

14/09/04 10:10:55 INFOmapred. JobClient: CPU time spent (MS) = 120

14/09/04 10:10:55 INFOmapred. JobClient: Total committedheap usage (bytes) = 15925248

14/09/04 10:10:55 INFOmapred. JobClient: Virtual memory (bytes) snapshot = 377499648

14/09/04 10:10:55 INFOmapred. JobClient: Combine inputrecords = 0

14/09/04 10:10:55 INFOmapred. JobClient: Reduce inputrecords = 0

Hadoop @ HADOOP :~ $

 

 

Display result

Hadoop @ HADOOP :~ $ Hadoop fs-ls output

Warning: $ HADOOP_HOME is deprecated.

 

Found 3 items

-Rw-r -- 1 hadoop supergroup 02014-09-04 10:10/user/hadoop/output/_ SUCCESS

Drwxr-xr-x-hadoop supergroup 02014-09-04 10: 10/user/hadoop/output/_ logs

-Rw-r -- 1 hadoop supergroup 02014-09-04/user/hadoop/output/part-r-00000

Hadoop @ HADOOP :~ $

 

View execution results

Hadoop @ HADOOP :~ $ Hadoop fs-cat output/part-r-00000

Hadoop 1

Hello 2

World 1

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition

Build a Hadoop environment (using virtual machines to build two Ubuntu systems in a Winodws environment)