Full distributed model of Hadoop Environment building

First, this article explains:
This test is fully distributed on five virtual machine systems. Hadoop distributed mode is distributed on multiple machines, configuration is very similar to pseudo-distributed, the name node and data node are distributed on different hosts, each data node has the corresponding task scheduling.

Second, the environment description:
Virtual Software: VM10.0
Operating system: Cent OS 6.7
Hadoop version: hadoop-2.7.2
JDK version: jdk1.8_77
Note: Built on an independent model, refer to the article: the standalone model of the Hadoop environment.

Three, the basic Environment configuration
1. Set up different configuration folders for different modes under ${hadoop_home}/etc/
Standalone mode: Hadoop_alone (recommended)
Pseudo distribution mode: Hadoop_presudo (recommended)
Fully distributed mode: Hadoop_cluster (recommended)

2, the parameter configuration directory to specify the way
A, default mode: ${hadoop_home}/etc/hadoop/*.xml
B. Specify the configuration directory through the startup parameters
Start-dfs.sh–config=/path
C, through the environment variable Hadoop_conf_dir=/path (/etc/profile)
Note: Because of the full distribution pattern: I am here to format the name node and the B method used when starting Hadoop.
Iv. Environment Construction
1th Step: Build the Java and Hadoop infrastructure on the master machine.
Reference article: Building a Hadoop environment in standalone mode

2nd step: Configure SSH on the master machine
650) this.width=650; "src=" Http://www.itroad.org/uploadfile/2016/0414/20160414065505551.png "style=" height:370px; width:600px; "alt=" 20160414065505551.png "/>

3rd Step: Configure the hosts on the master machine
650) this.width=650; "src=" Http://www.itroad.org/uploadfile/2016/0414/20160414065538112.png "style=" HEIGHT:153PX; width:600px; "alt=" 20160414065538112.png "/>


4th Step: Clone 4 virtual machines and modify the configuration of the 4 network cards
Reference article: How to modify network adapter after cloning Linux

5th step: Configure ${hadoop_home}/etc/hadoop_ cluster/core-site.xml 650) this.width=650 on the master machine; "Src=" http://www.itroad.org/ Uploadfile/2016/0414/20160414065602348.png "style=" height:436px;width:600px; "alt=" 20160414065602348.png "/>


6th Step: Configure ${hadoop_home}/etc/hadoop_cluster/hdfs-site.xml
650) this.width=650; "src=" Http://www.itroad.org/uploadfile/2016/0414/20160414065621844.png "style=" HEIGHT:414PX; width:600px; "alt=" 20160414065621844.png "/>


7th Step, Configuration ${hadoop_home}/etc/hadoop_cluster/mapred-site.xml
650) this.width=650; "src=" Http://www.itroad.org/uploadfile/2016/0414/20160414070011229.png "style=" HEIGHT:464PX; width:600px; "alt=" 20160414070011229.png "/>



8th step, Configuration ${hadoop_home}/etc/hadoop_cluster/yarn-site.xml 650) this.width=650; "Src=" Http://www.itroad.org/uploadfile /2016/0414/20160414070045314.png "style=" height:479px;width:600px; "alt=" 20160414070045314.png "/>


9th step, delete the ${hadoop_home}/etc/hadoop_cluster on the other four hosts

The 10th step, the machine on the ${hadoop_home}/etc/hadoop_cluster remote copy to another four machines on the ${hadoop_home}/etc/
650) this.width=650; "src=" Http://www.itroad.org/uploadfile/2016/0414/20160414065737404.png "style=" HEIGHT:41PX; width:600px; "alt=" 20160414065737404.png "/>

etc...

11th step, format the HDFs file system on the master machine
650) this.width=650; "src=" Http://www.itroad.org/uploadfile/2016/0414/20160414070118629.png "style=" height:30px; width:600px; "alt=" 20160414070118629.png "/>


12th Step, start the daemon for HDFs and Yran (start-dfs.sh,start-yarn.sh)
start-dfs.sh--config/home/program/hadoop/etc/hadoop_cluster/
start-yarn.sh--config/home/program/hadoop/etc/hadoop_cluster/
Note the stop process is followed by yarn close then close HDFs, do not format the name node frequently

13th step, operation of the files in HDFs via FS
A. Create directory: Hadoop fs-mkdir-p/user/ubuntu
B. Display directory: Hadoop fs-ls-r/
......
14th step, visit: http://IP:50070
Visit: http://IP:8088
650) this.width=650; "src=" Http://www.itroad.org/uploadfile/2016/0414/20160414070141437.png "style=" HEIGHT:424PX; width:600px; "alt=" 20160414070141437.png "/>
650) this.width=650; "src=" Http://www.itroad.org/uploadfile/2016/0414/20160414070144910.png "style=" HEIGHT:207PX; width:600px; "alt=" 20160414070144910.png "/>

Full distributed model of Hadoop Environment building