As Linux and Hadoop small white, the initial installation of Hadoop is really tired, took two days to install successfully. Below is a description of the Hadoop installation:
first, machine configuration and installation instructions
A laptop, VMware, I7 processor. Centos-7 System. Hadoop uses 2.7.2
Ii. installation of three nodes with VMware。 (Here you can build a CentOS virtual machine, because you need to install JDK,HADOOP, and configure environment variables, you can use the virtual machine cloning method to get another two nodes, save a lot of time and trouble.) Of course, the hostname of the different nodes will be modified later. Recommend this option! ）
1, the first is to install the JDK. There is not much to say, online a lot, I use the jdk-1.8.0_77
2, according to the basic requirements of the Hadoop cluster, one of which is the master node, mainly used to run Namenode, Secondorynamenode and ResourceManager tasks in Hadoop programs. The other two nodes are slave nodes, one of which is for redundancy purposes, if there is no redundancy, it can not be called Hadoop, so the simulation Hadoop cluster at least 3 nodes, if the computer configuration is very high, you can consider adding some other nodes. Slave nodes will primarily run Datanode and NodeManager tasks in Hadoop programs.
Therefore, after you have prepared these 3 nodes, you need to rename the host name of the Linux system (because the copy and paste operations are preceded by the other two nodes, at which point the host name of the 3 nodes is the same), rename the host name method:
Command: Vi/etc/hostname (fully modified)
By modifying the hostname file, these three points should be modified to differentiate.
Here are my three nodes of Ubuntu system host named: Master, Node1, Node2.
three, each Linux virtual machine configuration static IP
Command: cd/etc/sysconfig/network-scripts/
VI ifcfg-eth1
Modify DHCP to static, add IP address, MAC address, subnet mask, gateway, etc. Online Centos7 a lot.
Iv. Modify the Hosts file.
Command: vi/etc/hosts
Master IP address (192.***.***.***)
Node1 IP address ()
Node2 IP address ()
V. Establish a Hadoop running account (This step is what many people should do when building a Linux system)
sudo groupadd Hadoop//Set up Hadoop user groups
sudo useradd–s/bin/bash–d/home/hadoop–m hadoop–g hadoop–g admin//Add a ZHM user, this user belongs to the Hadoop user group and has admin privileges.
sudo passwd Hadoop//Set user ZHM login password
Su Hadoop//switch to ZHM users
Six, configure SSH password-free access.
No password login, the effect is on the master, through SSH node1 or ssh node2 can log on to the other computer. And you don't have to enter a password.
1. On three virtual machines, use ssh-keygen-t RSA and press ENTER on the line. This is the main place to set the SSH key and key storage path. The path is ~/.ssh. Open ~/.ssh There are three files: Authorizedkeys (Certified keys), Idrsa (private key), id_rsa.pub (public key)
2. On master, place the public key in the Authorizedkeys. Command: sudo cat Idrsa.pub>>authorized_keys
3. Place the Authorized_keys on master in the ~/.ssh directory of Node1 and Node2.
Command: sudo scp authorized_keys [email protected]:~/.ssh
Usage: sudo SCP authorized_keys remote host user name @ remote host name or IP: Store path.
4. Modify Authorizedkeys permissions, command: chmod 644 Authorizedkeys
5. Whether the test was successful
SSH Node1 Enter the user name password, and then exit, SSH node1 without password, directly into the system. That means it's a success.
6.exit exit back to the original node.
Vii. download and unzip the Hadoop installation package
The Hadoop installation package is placed under the/home/hadoop
Command: TAR–ZXVF hadoop-2.7.2.tar.gz #解压hadoop安装包
Configuration path after successful installation: similar to JDK.
Command: Vi/etc/profile Eight, configure Namenode modify site file
1. Modify core-site.xml File command: Cd/home/hadoop/hadoop-2.7.2/etc/hadoop ls can see core-site.xml hdfs-site.xml yarn -site.xml command: VI core-site.xml 2, modify hdfs-site.xml File command: VI hdfs-site.xml 3, modify the Yarn-site.xml file 4, modify Mapred-site.xml.template 5, modify hadoop-env.sh File command: Source hadoop-env.sh Let it take effect
Ix. Configuration Slaves File command: Cd/home/hadoop/hadoop-2.7.2/etc/hadoop ls view slaves file VI slaves 10, to each section Point replication Hadoop does not require this step because it is configured individually. If you are configuring only one file and environment that can use the SCP command to replicate the Hadoop configuration to the other two nodes Xi. format namenode, format only once
Command: Hadoop namenode-format
If successfully format is present, Success!!
12. Start Hadoop
Cut to the/home/hadoop/hadoop-2.7.2/sbin directory, command:./start-all.sh
View the boot daemon at the Node1 and Node2 nodes: JPS
At this point, you are done!!
hadoop2.7.2 fully distributed installation based on CentOS
