Install hadoop1.2.1 Cluster Environment and hadoop1.2.1 Cluster Environment
I. Planning (I) hardware resources 10.252.48.155 slave15510.252.48.156 slave15610.252.48.166 master166
(2) Basic information user: root/jediael/directory:/opt/jediael/
II. Environment configuration (1) Unified user name and password, and grant jediael the permission to execute all commands [root @ irms155 ~] # Passwd [root @ irms155 ~] # Useradd jediael
[Root @ irms155 ~] # Passwd jediael
[Root @ irms155 ~] # Add the following line to vi/etc/sudoers: jediael ALL = (ALL) ALL
(2) create a directory/opt/jediael [jediael @ irms155 ~] $ Cd/opt
[Jediael @ irms155 opt] $ sudo mkdir jediael [jediael @ irms155 opt] $ sudo chown jediael: jediael/
(3) modify the username and/etc/hosts file 1. Modify/etc/sysconfig/networkNETWORKING = yesHOSTNAME = ******** 2. Modify/etc/hosts10.252.48.155 slave15510.252.48.156 prepare master166 pay attention to the hosts file you cannot configure 127.0.0.1, otherwise, an exception occurs. Org. apache. hadoop. ipc. Client: Retrying connect to server: master166/10.252.48.166: 9000. Already trie
(4) Configure password-free logon and run the preceding command on master166: $ ssh-keygen-t dsa-p'-f ~ /. Ssh/id_dsa
$ Cat ~ /. Ssh/id_dsa.pub> ~ /. Ssh/authorized_keys and then copy authorized_keys to slave155, slave156scp authorized_keys slave155 :~ /. Ssh/scp authorized_keys slave155 :~ /. Ssh/
Note: (1) if the. ssh directory does not exist, create a directory first. (2) The ssh/permission is 600, and the authorized_keys permission is 700. If the permission is too large, it will not work.
(5) install java, and set the relevant environment variables reference http://blog.csdn.net/jediael_lu/article/details/38925871 ( download hadoop-1.2.1.tar.gz, and decompress it to/opt/jediael
3. modify the configuration file (1) Modify conf/hadoop_env.shexport JAVA_HOME =/usr/java/jdk1.7.0 _ 51 (2) modify the core-site.xml <property> <name> fs. default. name </name> <value> hdfs: // master166: 9000 </value> </property> <name> hadoop. tmp. dir </name> <value>/tmp </value> </property>
(3) Modify hdfs-site.xml <property> <name> dfs. replication </name> <value> 2 </value> </property>
(4) Modify mapred-site.xml <property> <name> mapred. job. tracker </name> <value> master166: 9001 </value> </property>
(5) Modify master and slavesmaster: master166
Slaves: slave155slave156
4. Start and verify
1. formatized namenode [jediael @ master166 hadoop-1.2.1] $ bin/hadoop namenode-format
14/08/29 15:11:45 INFO namenode. NameNode: STARTUP_MSG:
/*************************************** *********************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = master166/127.0.0.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2-r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.7.0 _ 51
**************************************** ********************/
14/08/29 15:11:45 INFO util. GSet: Computing capacity for map BlocksMap
14/08/29 15:11:45 INFO util. GSet: VM type = 64-bit
14/08/29 15:11:45 INFO util. GSet: 2.0% max memory = 932184064
14/08/29 15:11:45 INFO util. GSet: capacity = 2 ^ 21 = 2097152 entries
14/08/29 15:11:45 INFO util. GSet: recommended = 2097152, actual = 2097152
14/08/29 15:11:46 INFO namenode. FSNamesystem: fsOwner = jediael
14/08/29 15:11:46 INFO namenode. FSNamesystem: supergroup = supergroup
14/08/29 15:11:46 INFO namenode. FSNamesystem: isPermissionEnabled = true
14/08/29 15:11:46 INFO namenode. FSNamesystem: dfs. block. invalidate. limit = 100
14/08/29 15:11:46 INFO namenode. FSNamesystem: isAccessTokenEnabled = false accessKeyUpdateInterval = 0 min (s), accessTokenLifetime = 0 min (s)
14/08/29 15:11:46 INFO namenode. FSEditLog: dfs. namenode. edits. toleration. length = 0
14/08/29 15:11:46 INFO namenode. NameNode: Caching file names occuring more than 10 times
14/08/29 15:11:46 INFO common. Storage: Image file/tmp/dfs/name/current/fsimage of size 113 bytes saved in 0 seconds.
14/08/29 15:11:46 INFO namenode. FSEditLog: closing edit log: position = 4, editlog =/tmp/dfs/name/current/edits
14/08/29 15:11:46 INFO namenode. FSEditLog: close success: truncate to 4, editlog =/tmp/dfs/name/current/edits
14/08/29 15:11:46 INFO common. Storage: Storage directory/tmp/dfs/name has been successfully formatted.
14/08/29 15:11:46 INFO namenode. NameNode: SHUTDOWN_MSG:
/*************************************** *********************
SHUTDOWN_MSG: Shutting down NameNode at master166/127.0.0.1
**************************************** ********************/
2. Start hadoop [jediael @ master166 hadoop-1.2.1] $ bin/start-all.sh
Starting namenode, logging to/opt/jediael/hadoop-1.2.1/libexec/../logs/hadoop-jediael-namenode-master166.out
Slave155: starting datanode, logging to/opt/jediael/hadoop-1.2.1/libexec/../logs/hadoop-jediael-datanode-slave155.out
Slave156: starting datanode, logging to/opt/jediael/hadoop-1.2.1/libexec/.../logs/hadoop-jediael-datanode-slave156.out
Master166: starting secondarynamenode, logging to/opt/jediael/hadoop-1.2.1/libexec/../logs/hadoop-jediael-secondarynamenode-master166.out
Starting jobtracker, logging to/opt/jediael/hadoop-1.2.1/libexec/../logs/hadoop-jediael-jobtracker-master166.out
Slave155: starting tasktracker, logging to/opt/jediael/hadoop-1.2.1/libexec/../logs/hadoop-jediael-tasktracker-slave155.out
Slave156: starting tasktracker, logging to/opt/jediael/hadoop-1.2.1/libexec/.../logs/hadoop-jediael-tasktracker-slave156.ou
3. Verify NameNode http: // 172.16.80.166: 50070 JobTracker http: // 172.16.80.166: 50030 on the logon page.
4. view the java Process of each host (1) master166: [jediael @ master166 hadoop-1.2.1] $ jps
17963 NameNode
18280 JobTracker
Jps 18446
18171 SecondaryNameNode (2) slave155: [jediael @ slave155 hadoop-1.2.1] $ jps
Jps 16019
15858 DataNode
15954 TaskTracker (3) slave156: [jediael @ slave156 hadoop-1.2.1] $ jps
Jps 15625
15465 DataNode
15561 TaskTracker
5. Run a complete mapreduce program.
The following content is only executed on the master 1. Copy the wordcount. jar package to the server for the program see the http://blog.csdn.net/jediael_lu/article/details/37596469
2. Create an input directory and copy the related files to the directory [jediael @ master166 ~] $ Hadoop fs-mkdir/wcin [jediael @ master166 projects] $ hadoop fs-copyFromLocal/opt/jediael/hadoop-1.2.1/conf/hdfs-site.xml/wcin
3. Run the program [jediael @ master166 projects] $ hadoop jar wordcount. jar org. jediael. hadoopdemo. wordcount. WordCount/wcin/wcout
14/08/31 20:04:26 WARN mapred. JobClient: Use GenericOptionsParser for parsing the arguments. Applications shocould implement Tool for the same.
14/08/31 20:04:26 INFO input. FileInputFormat: Total input paths to process: 1
14/08/31 20:04:26 INFO util. NativeCodeLoader: Loaded the native-hadoop library
14/08/31 20:04:26 WARN snappy. LoadSnappy: Snappy native library not loaded
14/08/31 20:04:26 INFO mapred. JobClient: Running job: job_201408311554_0003
14/08/31 20:04:27 INFO mapred. JobClient: map 0% reduce 0%
14/08/31 20:04:31 INFO mapred. JobClient: map 100% reduce 0%
14/08/31 20:04:40 INFO mapred. JobClient: map 100% reduce 100%
14/08/31 20:04:40 INFO mapred. JobClient: Job complete: job_201408311554_0003
14/08/31 20:04:40 INFO mapred. JobClient: Counters: 29
14/08/31 20:04:40 INFO mapred. JobClient: Job Counters
14/08/31 20:04:40 INFO mapred. JobClient: Launched reduce tasks = 1
14/08/31 20:04:40 INFO mapred. JobClient: SLOTS_MILLIS_MAPS = 4230
14/08/31 20:04:40 INFO mapred. JobClient: Total time spent by all CES waiting after reserving slots (MS) = 0
14/08/31 20:04:40 INFO mapred. JobClient: Total time spent by all maps waiting after reserving slots (MS) = 0
14/08/31 20:04:40 INFO mapred. JobClient: Launched map tasks = 1
14/08/31 20:04:40 INFO mapred. JobClient: Data-local map tasks = 1
14/08/31 20:04:40 INFO mapred. JobClient: SLOTS_MILLIS_REDUCES = 8531
14/08/31 20:04:40 INFO mapred. JobClient: File Output Format Counters
14/08/31 20:04:40 INFO mapred. JobClient: Bytes Written = 284
14/08/31 20:04:40 INFO mapred. JobClient: FileSystemCounters
14/08/31 20:04:40 INFO mapred. JobClient: FILE_BYTES_READ = 370
14/08/31 20:04:40 INFO mapred. JobClient: HDFS_BYTES_READ = 357
14/08/31 20:04:40 INFO mapred. JobClient: FILE_BYTES_WRITTEN = 104958
14/08/31 20:04:40 INFO mapred. JobClient: HDFS_BYTES_WRITTEN = 284
14/08/31 20:04:40 INFO mapred. JobClient: File Input Format Counters
14/08/31 20:04:40 INFO mapred. JobClient: Bytes Read = 252
14/08/31 20:04:40 INFO mapred. JobClient: Map-Reduce Framework
14/08/31 20:04:40 INFO mapred. JobClient: Map output materialized bytes = 370
14/08/31 20:04:40 INFO mapred. JobClient: Map input records = 11
14/08/31 20:04:40 INFO mapred. JobClient: Reduce shuffle bytes = 370
14/08/31 20:04:40 INFO mapred. JobClient: Spilled Records = 40
14/08/31 20:04:40 INFO mapred. JobClient: Map output bytes = 324
14/08/31 20:04:40 INFO mapred. JobClient: Total committed heap usage (bytes) = 238026752
14/08/31 20:04:40 INFO mapred. JobClient: CPU time spent (MS) = 1130
14/08/31 20:04:40 INFO mapred. JobClient: Combine input records = 0
14/08/31 20:04:40 INFO mapred. JobClient: SPLIT_RAW_BYTES = 105
14/08/31 20:04:40 INFO mapred. JobClient: Reduce input records = 20
14/08/31 20:04:40 INFO mapred. JobClient: Reduce input groups = 20
14/08/31 20:04:40 INFO mapred. JobClient: Combine output records = 0
14/08/31 20:04:40 INFO mapred. JobClient: Physical memory (bytes) snapshot = 289288192
14/08/31 20:04:40 INFO mapred. JobClient: Reduce output records = 20
14/08/31 20:04:40 INFO mapred. JobClient: Virtual memory (bytes) snapshot = 1533636608
14/08/31 20:04:40 INFO mapred. JobClient: Map output records = 20
4. view results
[Jediael @ master166 projects] $ hadoop fs-cat/wcout /*
--> 1
<! -- 1
</Configuration> 1
</Property> 1
<? Xml 1
<? Xml-stylesheet 1
<Configuration> 1
<Name> dfs. replication </name> 1
<Property> 1
<Value> 2 </value> 1
Put 1
File. 1
Href = "configuration. xsl"?> 1
In 1
Overrides 1
Property 1
Site-specific 1
This 1
Type = "text/xsl" 1
Version = "1.0"?> 1
Cat: File does not exist:/wcout/_ logs
Cluster Environment
The Cluster Environment of CAS, including the cluster environment for CAS customer applications and the Cluster Environment for CAS services. To use CAS in a cluster environment, we need to solve two problems: first, how does CAS correctly forward exit requests to the specific client application server where the user session is located during single-point exit, instead of forwarding to other cluster servers, the second is to share various Ticket information in the CAS server cluster environment.
Hadoop video tutorial download?
It seems that there is no free service yet, and there are not many charges. The price is too high! The threshold for this item is relatively high. I only have one computer, so it cannot be done! I have done this again recently! Watching videos can help you better understand the concept! Read more books and practice more!
