"Basic Hadoop Tutorial" 4, Full distributed environment construction of Hadoop

Last blog we completed the construction of the Hadoop pseudo-distributed environment, pseudo-distributed mode is also called single node cluster mode, NameNode, Secondarynamenode, DataNode, Jobtracker, Tasktracker all daemons run on top of the K-master node. In this blog we will build a fully distributed environment, running Namenode, Secondarynamenode, Jobtracker daemon on the main node, running Datanode, tasktracker on the slave node.

Development environment

Hardware environment: Centos 6.5 server 4 (one for master node, three for slave node)
Software Environment: Java 1.7.0_45, hadoop-1.2.1

1. Cluster topology diagram

We use 4 machines to build a fully distributed Hadoop environment, and the topology diagram for 4 machines is as follows:

The roles for each node in the Hadoop cluster are shown in the following table:

2. Configure SSH

• Environment preparation

The following to configure the K-master start SSH service as an example of the demonstration, the user needs to follow the steps below to complete the Kvmslave1~kvmslave3 three sub-node machine SSH service start;

1) Start the SSH service with the Hadoop user telnet to the K-master server;

[[email protected] hadoop]$ sudo /etc/init.d/sshd start

2) Set boot up:

[[email protected] hadoop]$ sudo chkconfig sshd on

• Configure K-master native No password login

Below to configure k-master this machine without password login As an example to explain, the user should refer to the following steps to complete the Kvmslave1~kvmslave3 three table node machine of the native no password login;

1) A Hadoop user telnet to the K-master server, generate a public and private key password pair on the K-master server, the key is stored in the "/home/hadoop/.ssh" directory by default, generate public and private key password pair, no need to enter a password, directly enter.

#切换到hadoop用户，不能使用root用户生成密钥[[email protected] hadoop]$ su hadoop[[email protected] hadoop]$ ssh-keygen -t rsa -P ""

2) Append the public key to the "Authorized_keys" file

[[email protected] hadoop]$ cd /home/hadoop/[[email protected] hadoop]$ cat .ssh/id_rsa.pub >> .ssh/authorized_keys

3) Giving permission

[[email protected] hadoop]$ chmod 600 .ssh/authorized_keys

4) Verify that this function has no password access

[[email protected] hadoop]$ ssh K-Master

• Configure K-master native no password login kvmslave1~kvmslave3

below to K-master no password login KVMSlave1 as an example to explain, the user should refer to the following steps to complete k-master no password login kvmslave2~kvmslave3.

1) with Hadoop users telnet to the KVMSLAVE1 server, copy the K-master server's public key "Id_rsa.pub" to the KVMSLAVE1 server's "/home/hadoop/" directory.

[[email protected] hadoop]$ cd /home/hadoop/[[email protected] hadoop]$ scp [email protected]:/home/hadoop/.ssh/id_rsa.pub /home/hadoop/

2) Append the K-master public key (/home/hadoop/id_rsa.pub) to the Authorized_keys of KVMSlave1.

[[email protected] hadoop]$ cd /home/hadoop[[email protected] hadoop]$ cat id_rsa.pub >> .ssh/authorized_keys[[email protected] hadoop]$ rm –r /home/hadoop/id_rsa.pub

3) also open a terminal, Telnet k-master server, in the K-master server test through SSH without password login KVMSlave1.

[[email protected] hadoop]$ ssh KVMSlave1

• Configure KVMSLAVE1~KVMSLAVE3 native no password login k-master

below to KVMSlave1 no password login K-master as an example to explain, the user should refer to the following steps to complete kvmslave2~kvmslave3 no password login k-master.

1) with Hadoop users telnet to K-master, copy the KVMSLAVE1 server's public key "Id_rsa.pub" to the KVMSLAVE1 server's "/home/hadoop/" directory.

[[email protected] hadoop]$ scp [email protected]:/home/hadoop/.ssh/id_rsa.pub /home/hadoop

2) Append the KVMSlave1 public key (/home/hadoop/id_rsa.pub) to the Authorized_keys of K-master.

[[email protected] hadoop]$ cd /home/hadoop[[email protected] hadoop]$ cat id_rsa.pub >> .ssh/authorized_keys[[email protected] hadoop]$ rm –r /home/hadoop/id_rsa.pub

3) with Hadoop users telnet to the KVMSLAVE1 server, test the KVMSLAVE1 server with no password via SSH login k-master.

[[email protected] hadoop]$ ssh K-Master

3. Install Hadoop

If the user has completed the Hadoop pseudo-distributed environment, it is recommended to remove the/usr/hadoop/installation environment and configure the Hadoop fully distributed environment from scratch.

1) Telnet to the K-master server, download the hadoop-1.2.1.tar.gz, and copy it to the/home/hadoop/directory of the K-master server.

2) Extract the Hadoop source files

[[email protected] ~]$ su hadoop[[email protected] ~]$ cd /usr[[email protected] usr]$ sudo tar -zxvf  /home/hadoop/hadoop-1.2.1.tar.gz        //将文件减压在当前路径

3) Renaming Hadoop

[[email protected] usr]$ sudo mv hadoop-1.2.1/ hadoop/

4) Set up user group and user groups for Hadoop folders

It's critical to one step to make it easy for Hadoop users to have read and write access to the folder's files, or you won't be able to create files and write log information in that folder after subsequent Hadoop starts.

[[email protected] usr]$ sudo chown -R hadoop:hadoop /usr/hadoop

5) Remove the installation package

[[email protected] ~]$ rm –rf  /home/hadoop/hadoop-1.2.1.tar.gz#删除"hadoop-1.2.1.tar.gz"安装包

4. Configuring the K-master Hadoop environment

1) Configure Environment variables

Enables the Hadoop command to take effect immediately at the current terminal;

[h[email protected] ~] $source /etc/profile

2) configuration hadoop-env.sh

The Hadoop environment is based on the JVM virtual machine environment, so you need to specify the JDK environment in the hadoop-env.sh configuration file. Modify the/usr/hadoop/conf/hadoop-env.sh file to add the following JDK configuration information.

[[email protected] ~] cd /usr/hadoop/[[email protected] hadoop] vi conf/hadoop-env.shexport JAVA_HOME=/usr/java/jdk1.7.0_65

3) configuration Core-site.xml

Modify the Hadoop core profile/usr/hadoop/conf/core-site.xml, specifying the IP address and port number of the NameNode by Fs.default.name, Specify a temporary folder for the Hadoop data store through Hadoop.tmp.dir.

[[email protected] hadoop] vi conf/core-site.xml<configuration>    <property>        <name>fs.default.name</name>        <value>hdfs://K-Master:9000</value>    </property>    <property>        <name>hadoop.tmp.dir</name>        <value>/usr/hadoop/tmp</value>    </property></configuration>

Special Note: If the Hadoop.tmp.dir parameter is not configured, the system default temp directory is:/tmp/hadoo-hadoop. This directory is deleted after each reboot and must be re-executed, or it will fail.

4) Configuration Hdfs-site.xml

Modify the HDFs core configuration file/usr/hadoop/conf/ Hdfs-site.xml, specifying the backup factor of HDFs by Dfs.replication is 3, specifying the file storage directory of the Namenode node through Dfs.name.dir, and specifying the DFS.DATA.DIR node's file storage directory through Datanode.

[[email protected] hadoop] vi conf/hdfs-site.xml<configuration>    <property>        <name>dfs.replication</name>        <value>3</value>    </property>    <property>        <name>dfs.name.dir</name>        <value>/usr/hadoop/hdfs/name</value>    </property>    <property>        <name>dfs.data.dir</name>        <value>/usr/hadoop/hdfs/data</value>    </property></configuration>

5) Configuration Mapred-site.xml

Modify the MapReduce core profile/usr/hadoop/conf/mapred-site.xml, specifying the address and port of Jobtracker through the Mapred.job.tracker property.

[[email protected] hadoop] vi conf/mapred-site.xml<configuration>    <property>        <name>mapred.job.tracker</name>        <value>http://K-Master:9001</value>    </property></configuration>

6) Configure the Masters file

Modify the/user/hadoop/conf/masters file that specifies the server machine where the Namenode node resides. Remove localhost, add the hostname of the Namenode node K-master; IP address is not recommended because the IP address may change, but the hostname generally does not change.

[[email protected] hadoop] vi conf/mastersK-Master

7) Configure the Slaves file (master host-specific)

Modifies the/usr/hadoop/conf/slaves file, which specifies which server nodes are Datanode nodes. Remove Locahost and add the hostname of all the Datanode nodes as shown below.

[[email protected] hadoop] vi conf/slavesKVMSlave1KVMSlave2KVMSlave3

5. Configuring the Kvmslave Hadoop environment

The following is a demonstration of the Hadoop configuration KVMSlave1, which the user needs to complete the configuration of the other Kvmslave server, using the following steps.

1) with Hadoop users telnet to the KVMSLAVE1 server, copy the K-master host's Hadoop folder to the local/usr/directory;

[[email protected] ~]$ cd /usr/[[email protected] usr]$ sudo scp -r [email protected]:/usr/hadoop/ .[[email protected] usr]$ sudo chown -R hadoop:hadoop hadoop/#slaves文件内容删除，或者直接删除slaves[[email protected] usr]$ rm /usr/hadoop/conf/slaves

2) Configure Environment variables

Enables the Hadoop command to take effect immediately at the current terminal;

[[email protected] ~]$ source /etc/profile

6. Format HDFs File system

The format HDFs file system needs to be executed by the Hadoop user on the Namenode node, only once, and no further formatting is required for the next boot, and the HDFs file management system and the MapReduce service can be started directly.

[[email protected] ~]$ Hadoop namenode-format14/07/24 16:37:57 INFO namenode. Namenode:startup_msg:/************************************************************startup_msg:starting Namenodestartup_msg:host = K-master/192.168.100.147startup_msg:args = [-format]startup_msg:version = 1.2.1STARTUP _msg:build = Https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2-r 1503152; Compiled by ' Mattf ' on Mon Jul 15:23:09 PDT 2013startup_msg:java = 1.7.0_65********************a******************** /14/07/24 16:37:57 INFO util. Gset:computing capacity for map blocksmap14/07/24 16:37:57 INFO util. GSET:VM type = 64-bit14/07/24 16:37:57 INFO util. gset:2.0% max memory = 93218406414/07/24 16:37:57 INFO util. gset:capacity = 2^21 = 2097152 entries14/07/24 16:37:57 INFO util. gset:recommended=2097152, actual=209715214/07/24 16:37:58 INFO namenode. FSNAMESYSTEM:FSOWNER=HADOOP14/07/24 16:37:58 INFO Namenode. fsnamesystem:supergroup=supergroup14/07/24 16:37:58 INFO Namenode. FSNAMESYSTEM:ISPERMISSIONENABLED=TRUE14/07/24 16:37:58 INFO Namenode. FSNAMESYSTEM:DFS.BLOCK.INVALIDATE.LIMIT=10014/07/24 16:37:58 INFO Namenode. Fsnamesystem:isaccesstokenenabled=false accesskeyupdateinterval=0 min (s), Accesstokenlifetime=0 min (s) 14/07/24 16:37:58 INFO Namenode. FSEditLog:dfs.namenode.edits.toleration.length = 014/07/24 16:37:58 INFO namenode. namenode:caching file names occuring more than times 14/07/24 16:37:58 INFO Common. Storage:image file/usr/hadoop/hdfs/name/current/fsimage of size bytes saved in 0 seconds.14/07/24 16:37:59 INFO name Node. Fseditlog:closing edit log:position=4, editlog=/usr/hadoop/hdfs/name/current/edits14/07/24 16:37:59 INFO namenode. Fseditlog:close success:truncate to 4, editlog=/usr/hadoop/hdfs/name/current/edits14/07/24 16:37:59 INFO Common. Storage:storage Directory/usr/hadoop/hdfs/name has been successfully formatted.14/07/24 16:37:59 INFO Namenode. Namenode:shutdown_msg:/**************************Shutdown_msg:shutting down NameNode at k-master/192.168.100.147***************** *******************************************/

7. Start HDFs File Management system

1) Start the HDFs file management system through the start-dfs.sh command, start the Namenode node (k-master), Datanode node (KVMSlave1, KVMSlave2, KVMSLAVE3) and Secondarynamenode nodes (K-master).

[[email protected] ~]$ start-dfs.shstarting namenode, logging to /usr/hadoop/libexec/../logs/hadoop-hadoop-namenode-K-Master.outKVMSlave1: starting datanode, logging to /usr/hadoop/libexec/../logs/hadoop-hadoop-datanode-KVMSlave1.outKVMSlave2: starting datanode, logging to /usr/hadoop/libexec/../logs/hadoop-hadoop-datanode-KVMSlave2.outKVMSlave3: starting datanode, logging to /usr/hadoop/libexec/../logs/hadoop-hadoop-datanode-KVMSlave3.outK-Master: starting secondarynamenode, logging to /usr/hadoop/libexec/../logs/hadoop-hadoop-secondarynamenode-K-Master.out

2) View the startup process on the K-master node

If the following log information is printed, it indicates that the Namenode node initiated the Namenode and SecondaryNameNode2 service processes, that is, the Namenode node HDFs file management system started successfully.

[[email protected] ~]$ jps6164 Jps5971 NameNode6108 SecondaryNameNode

3) View the startup process on the KVMSLAVE1 node

Log on to the KVMSLAVE1 server with a Hadoop user and view the startup process through the JPS command. If the following log information is printed, it indicates that the Datanode node initiated the Datanode service process, that is, the KVMSLAVE1 node HDFs file management system started successfully, the user can be remotely logged on to other Datanode nodes to see if their HDFs file management system started successfully.

[[email protected] ~]$ jps1327 Jps1265 DataNode

8. Start MapReduce

1) through the start-mapred.sh command to start the MapReduce distributed computing service, after startup through the following log information can be seen on the Namenode node started the jobtracker process, respectively, in the Datanode node (KVMSlave1, The Tasktracker process was initiated on KVMSlave2, KVMSlave3).

[[email protected] ~]$ start-mapred.shstarting jobtracker, logging to /usr/hadoop/libexec/../logs/hadoop-hadoop-jobtracker-K-Master.outKVMSlave1: starting tasktracker, logging to /usr/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-KVMSlave1.outKVMSlave2: starting tasktracker, logging to /usr/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-KVMSlave2.outKVMSlave3: starting tasktracker, logging to /usr/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-KVMSlave3.out

2) View the startup process on the K-master node

If the following log information is printed, it indicates that the Jobtracker process was newly initiated on the Namenode node, that is, the Namenode node Jobtracker started successfully.

[[email protected] ~]$ jps1342 NameNode1726 Jps1619 JobTracker1480 SecondaryNameNode

3) View the startup process on the KVMSLAVE1 node

Log on to the KVMSLAVE1 server with a Hadoop user and view the startup process through the JPS command. If the following log information is printed, it indicates that the Tasktracker process was newly initiated on the KVMSLAVE1 node, that is, the KVMSLAVE1 node Tasktracker started successfully. Users can remotely log on to other DATANODE nodes to see if Tasktracker started successfully.

[[email protected] ~]$ jps1549 TaskTracker1265 DataNode1618 Jps

9. Command to view the status of the Hadoop cluster

With the Simple JPS command, although you can view the HDFs file management system, the MapReduce service has started successfully, but cannot see the running state of the entire Hadoop cluster. We can view it through the Hadoop dfsadmin-report. This command allows you to quickly locate which nodes are dead, the capacity of HDFs and how much is used, and the hard disk usage of each node.

[[email protected] ~]$ Hadoop dfsadmin-reportconfigured capacity:238417846272 (222.04 GB) Present Capacity: 219128426496 (204.08 GB) DFS remaining:218227326976 (203.24 GB) Dfs used:901099520 (859.36 MB) Dfs used%: 0.41%under replic Ated blocks:72blocks with corrupt replicas:0missing blocks:0------------------------------------------------- Datanodes Available:3 (3 total, 0 dead) name:192.168.100.144:50010decommission status:normalconfigured capacity:794726 15424 (74.01 GB) Dfs used:300367872 (286.45 MB) Non DFS used:6218309632 (5.79 GB) DFS remaining:72953937920 (67.94 GB) DFS U  sed%: 0.38%dfs remaining%: 91.8%last contact:tue Feb 16:50:00 CST 2015name:192.168.100.148:50010decommission Status: Normalconfigured capacity:79472615424 (74.01 GB) Dfs used:300367872 (286.45 MB) Non dfs used:6242603008 (5.81 GB) Dfs Rem aining:72929644544 (67.92 GB) DFS used%: 0.38%dfs remaining%: 91.77%last contact:tue Feb 16:49:59 CST 2015name:192.168 .100.146:50010decommission Status:normalConfigured capacity:79472615424 (74.01 GB) Dfs used:300363776 (286.45 MB) Non dfs used:6828507136 (6.36 GB) DFS Remaining: 72343744512 (67.38 GB) DFS used%: 0.38%dfs remaining%: 91.03%last contact:tue Feb 16:50:00 CST 2015

You may like

Hadoop's server Infrastructure setup
Hadoop stand-alone mode Setup
Hadoop's pseudo-distributed environment building
Fully distributed environment for Hadoop construction

"Basic Hadoop Tutorial" 4, Full distributed environment construction of Hadoop