Hadoop Learning Notes (i)-HADOOP2.5.2+ZOOKEEPER3.6.4+CENTOSX64+VMWARE11 environment construction

Preface

Big data is really too hot, has been full of yearning for new technologies, in fact, Hadoop is not a new technology, has been for several years. Plan to learn the content of the entire Hadoop ecosystem. This article introduces the environment to build, oneself take the environment to grope for several days connected, finally build complete, record a bit. And share, if one day you also need, we share together. In this hope I can seriously finish every blog, insist on writing.

                                                                                                                                                 --chaosju

Environment + Tool preparation

Big data is really too hot, has been full of yearning for new technologies, in fact, Hadoop is not a new technology, has been for several years. Plan to learn the content of the entire Hadoop ecosystem. This article introduces the environment to build, oneself take the environment to grope for several days connected, finally build complete, record a bit. And share, if one day you also need, we share together. In this hope I can seriously finish every blog, insist on writing.

1.vmare Virtual Machine

2.xshell or SECURECRT

3.jdk1.6 or 1.7-------not recommended 1.8

ISO for 4.centos

5. Install SSH

6.hadoop2.5.2

Download:http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.5.2/hadoop-2.5.2.tar.gz

Download:http://hadoop.apache.org/releases.html#19+november%2c+2014%3a+release+2.5.2+available

7.zookeeper-3.4.6.tar

download:http://www.apache.org/dyn/closer.cgi/zookeeper/

Note : The hadoop-2.5.2 installation package provided by Apache is compiled on a 32-bit operating system because Hadoop relies on some C + + local libraries, So if you install hadoop-2.5.2 on a 64-bit operation, you need to re-compile on the 64 operating system (it is recommended to install the 32-bit system for the first time).

Compilation Tutorial: http://blog.csdn.net/ggz631047367/article/details/42460589

64-bit Hadoop2.5.2:http://pan.baidu.com/s/1ktnielt

Hadoop Build Environment preparation steps Like the virtual machine installation, the installation of Xshell is used, it is not here to say. To put it bluntly, I set up a fully distributed, 4 virtual machines, not one installation, with vwware cloning features, such as your JDK has been built, you can clone.

4 virtual machines are now installed with assumptions. Build a real start to the environment.

0. If you have 3 virtual machines with VMware clones, you need to ensure that the IP can be automatically obtained, make the following modifications. Where the cloned host needs to be modified

the operating system after cloning needs to reassign MAC address    A,vi /etc/sysconfig/network-scripts/ifcfg-eth0   (NIC information)    Remove two lines from the file: UUID and Physical address     -rf/etc/udev/rules.d/-persistent-net.rules file    C, restart   6 or reboot

Modify the above operation completed, the execution ifconfig can see the Etho IP, and the new MAC address

Ifconfig

1. Modify the Linux hostname and all 4 hosts modified

Vim/etc/sysconfig/network

Modify the hostname of each host, my name is Node1, Node2, Node3, Node4

2. Modify IP (optional) change to fixed ip,4 console all modifications

    vi /etc/sysconfig/network-scripts/ifcfg-eth0    #修改    bootproto=static    ipaddr =192.168. 239.6    # (The IP is the set solid-state IP)    NETMASK=255.255. 255.0     GATEWAY=192.168. 239.2

3. Modify the host name and IP mapping, all 4 hosts modified
Vim/etc/hosts

Authentication: ping hostname on each host separately, not IP authentication configured correctly

4. Turn off the firewall

Turn off the firewall, restart or auto restart
Service Iptables Stop
Permanently closed, it is recommended to permanently turn off chkconfig iptables off to view the firewall shutdown status

5.ssh Free Login

Configure Node1 to Node1-4 ssh-free login node1 on executionSsh-keygen-t RSASCP~/.SSH/id_rsa.pub node1:~/.SSH/SCP~/.SSH/id_rsa.pub node2:~/.SSH/SCP~/.SSH/id_rsa.pub node3:~/.SSH/SCP~/.SSH/id_rsa.pub node4:~/.SSH/Node1-Node4 is going to do it.Cat~/.SSH/id_rsa.pub >> ~/.SSH/Authorized_keys Configure Node2 to Node1 free loginSSH-copy-ID-I. Node1

Verification: SSH Node*, the first time you need to enter a password. And then you don't need it.

6. Install JDK, configure environment variables, etc.

I will not say this, I Baidu it

Cluster planning

Description: You have a certain understanding of the architecture of Hadoop, do not know it is OK, first set up to see

1. Namenode (NN) Node1, Node2

2. Datanode (DN) Node1, Node2

3. Zookeeper (ZK) Node1, Node2, Node3

3. Dfszkfailovercontroller (ZKFC) Node1, Node2

3. Journalnode (JN) Node2, Node3, Node4

3. ResourceManager (RM) Node1

3. DataManager (DM) Node2, Node3, Node4

install Deployment Zookeeper and Hadoop one , zookeeper installation configuration zookeeper cluster
1, Node1 on the decompression

tar -zxvf zookeeper-3.4. 6. Tar  Ln -sf/root/zookeeper-3.4. 6  /home/zk

2, modify the configuration on Node1

Node1 on: cd/home/zk/conf/CPzoo_sample.cfg zoo.cfg Vim zoo.cfg modified: DataDir=/opt/Zookeeper added at the end: server.1=node1:2888:3888server.2=node2:2888:3888server.3=NODE3:2888:3888then create a TMP foldermkdir/opt/Zookeeper Create an empty file againTouch/opt/zookeeper/myID finally writes the ID to the fileEcho 1>/opt/zookeeper/myid

3, the configuration of the zookeeper copy to Node2, Node3)

Copy the Zookeper installation files from Node1 to Node2 and Node3SCP-r/home/zk/node2:/RootSCP-r/home/zk/node3:/Root node2:LN-sf/root/zk/home/ZK node3:LN-sf/root/zk/home/ZK Copy the Zookeper configuration file on Node1 to Node2 and Node3SCP-r/opt/zookeeper/node2:/optSCP-r/opt/zookeeper/node3:/opt Note: Modify Node2, Node2 correspondence/opt/zookeeper/myID content Node2:Echo 2>/opt/zookeeper/myID node3:Echo 3>/opt/zookeeper/myid

Hadoop Learning Notes (i)-HADOOP2.5.2+ZOOKEEPER3.6.4+CENTOSX64+VMWARE11 environment construction