Hadoop2.7.2 fully distributed environment construction

Hadoop2.7.2 fully distributed environment construction

Share my Hadoop2.7.2 fully distributed environment setup process.

Server:
User: hadoop

Ip	Machine name	System
192.168.199.155	Master-hadoop	Ubuntu desktop 15
192.168.199.157	Slave1-hadoop	Ubuntu desktop 15
192.168.199.158	Slave2-hadoop	Ubuntu desktop 15

Java:
Version: 1.8.0 _ 73

Path:/usr/local/jdk

Hadoop:

• Version: 2.7.2
• Path:/usr/local/hadoop

--------------------------------------------------------------------------------

Start deployment

The following deployment actions are all performed on the master, and slave1 and slave2 are generated by cloning the virtual machine.

Set system environment variables, edit/etc/profile, and add at the end:

Modify hosts, edit/etc/hosts, and add at the end:

Configure ssh password-free Login

[Hadoop @ master-hadoop ~] $ Ssh-keygen-t rsa

[Hadoop @ slave1-hadoop ~] $ Cd/home/hadoop/. ssh/

[Hadoop @ slave1-hadoop. ssh] $ cat id_rsa.pub> authorized_keys

[Hadoop @ slave1-hadoop. ssh] $ chmod 600 authorized_keys

[Hadoop @ slave1-hadoop. ssh] $ chmod 700 ../. ssh/

Modify hadoop configuration files (modify node content based on hadoop paths in core-site.xml and hdfs-site.xml files)

Hadoop-env.sh

Slaves

Core-site.xml

Hdfs-site.xml

Yarn-site.xml

Mapred-site.xml

Clone the Virtual Machine and generate the slave node (after the vmware clone, the NIC mac should be re-generated, and then re-set the mac address in the ubuntu network settings)
1. Modify the machine name, edit/etc/hostname, and change the file content to slave1 | slave2 and then restart the system.
2. Connect slave1 and slave2 over ssh on the master to test whether password-free login is successful. Run
Ssh slave1-hadoop
3. Start hadoop on the master and execute
Start-all.sh

Note:
1. hadoop users must have/usr/local/hadoop folder Read and Write Permissions
2. ssh Login prompts The authenticity of host can't be established, modify The configuration of The/etc/ssh/ssh_config file, and add The following at The end:

StrictHostKeyChecking no

UserKnownHostsFile/dev/null

You may also like the following articles about Hadoop:

Tutorial on standalone/pseudo-distributed installation and configuration of Hadoop2.4.1 under Ubuntu14.04

Install and configure Hadoop2.2.0 on CentOS

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition