Hadoop cluster installation-CDH5 (three server clusters)

Source: Internet
Author: User

Hadoop cluster installation-CDH5 (three server clusters)

Hadoop cluster installation-CDH5 (three server clusters)

CDH5 package download: http://archive.cloudera.com/cdh5/

Host planning:

IP

Host

Deployment module

Process

192.168.107.82

Hadoop-NN-01

NameNode

ResourceManager

NameNode

DFSZKFailoverController

ResourceManager

192.168.107.83

Hadoop-DN-01

Zookeeper-01

DataNode

NodeManager

Zookeeper

DataNode

NodeManager

JournalNode

QuorumPeerMain

192.168.107.84

Hadoop-DN-02

Zookeeper-02

DataNode

NodeManager

Zookeeper

DataNode

NodeManager

JournalNode

QuorumPeerMain

Process description:

  • NameNode
  • ResourceManager
  • DFSZKFC: DFS Zookeeper Failover Controller activates Standby NameNode
  • DataNode
  • NodeManager
  • JournalNode: NameNode shares the editlog node service (if NFS sharing is used, the process and all startup configurations can be omitted ).
  • QuorumPeerMain: Zookeeper MAIN PROCESS

Directory planning:

Name

Path

$ HADOOP_HOME

/Home/hadoopuser/hadoop-2.6.0-cdh5.6.0

Data

$ HADOOP_HOME/data

Log

$ HADOOP_HOME/logs

 

Configuration:

1. Disable the firewall (the firewall can be configured later)

Ii. Install JDK (omitted)

3. Modify the HostName and configure the Host (3 hosts)

[root@Linux01 ~]# vim /etc/sysconfig/network[root@Linux01 ~]# vim /etc/hosts192.168.107.82 Hadoop-NN-01192.168.107.83 Hadoop-DN-01 Zookeeper-01192.168.107.84 Hadoop-DN-02 Zookeeper-01

4. Create a dedicated Hadoop login user (5) for security purposes)

[Root @ Linux01 ~] # Useradd hadoopuser [root @ Linux01 ~] # Passwd hadoopuser [root @ Linux01 ~] # Su-hadoopuser # Switch users

5. Configure SSH password-free Logon (2 NameNode servers)

[Hadoopuser @ Linux05 hadoop-2.6.0-cdh5.6.0] $ ssh-keygen # generate a public/private key [hadoopuser @ Linux05 hadoop-2.6.0-cdh5.6.0] $ ssh-copy-id-I ~ /. Ssh/id_rsa.pub hadoopuser @ Hadoop-NN-01

-I indicates input

~ /. Ssh/id_rsa.pub indicates which public key group

Or omitted:

[Hadoopuser @ Linux05 hadoop-2.6.0-cdh5.6.0] $ ssh-copy-id Hadoop-NN-01 (or write IP: 10.10.51.231) # throwing the public key to the opposite server [hadoopuser @ Linux05 hadoop-2.6.0-cdh5.6.0] $ ssh-copy-id "6000 Hadoop-NN-01" # Write this if there is a port

Note: Modify Hadoop's profile Hadoop-env.sh

Export HADOOP_SSH_OPTS = "-p 6000"

[Hadoopuser @ Linux05 hadoop-2.6.0-cdh5.6.0] $ ssh Hadoop-NN-01 # verify (exit current connection command: exit, logout) [hadoopuser @ Linux05 hadoop-2.6.0-cdh5.6.0] $ ssh Hadoop-NN-01-p 6000 # Write this if it has a port

6. Configure environment variables: vi ~ /. Bashrc and then source ~ /. Bashrc (5)

[Hadoopuser @ Linux01 ~] $ Vi ~ /. Bashrc # hadoop cdh5export HADOOP_HOME =/home/hadoopuser/hadoop-2.6.0-cdh5.6.0export PATH = $ PATH: $ HADOOP_HOME/sbin: $ HADOOP_HOME/bin [hadoopuser @ Linux01 ~] $ Source ~ /. Bashrc # effective

7. Install zookeeper (2 DataNode servers)

1. Extract

2. Configure environment variables: vi ~ /. Bashrc

[Hadoopuser @ Linux01 ~] $ Vi ~ /. Bashrc # zookeeper cdh5export ZOOKEEPER_HOME =/home/hadoopuser/zookeeper-3.4.5-cdh5.6.0export PATH = $ PATH: $ ZOOKEEPER_HOME/bin [hadoopuser @ Linux01 ~] $ Source ~ /. Bashrc # effective

3. Modify log output

[Hadoopuser @ Linux01 ~] $ Vi $ ZOOKEEPER_HOME/libexec/zkEnv. sh56: Find the following statement: ZOO_LOG_DIR = "$ ZOOKEEPER_HOME/logs"

4. modify the configuration file

[hadoopuser@Linux01 ~]$ vi $ZOOKEEPER_HOME/conf/zoo.cfg# zookeepertickTime=2000initLimit=10syncLimit=5dataDir=/home/hadoopuser/zookeeper-3.4.5-cdh5.6.0/dataclientPort=2181# clusterserver.1=Zookeeper-01:2888:3888server.2=Zookeeper-02:2888:3888

5. Set myid

(1) Hadoop-DN-01:

mkdir $ZOOKEEPER_HOME/dataecho 1 > $ZOOKEEPER_HOME/data/myid

(2) Hadoop-DN-02:

mkdir $ZOOKEEPER_HOME/dataecho 2 > $ZOOKEEPER_HOME/data/myid

6. Start each node:

[hadoopuser@Linux01 ~]$ zkServer.sh start

7. Verification

[hadoopuser@Linux01 ~]$ jps3051 Jps2829 QuorumPeerMain

8. Status

[hadoopuser@Linux01 ~]$ zkServer.sh statusJMX enabled by 

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.