1.linux Environment Configuration

First of all, here is the virtual machine environment.
1. Install Centos6.8-mini with VBox
Be careful not to use replication to install, the replicated virtual machine network does not pass
Install the following:

Host	IP	role	Memory
Hadoop1	192.168.0.10	Namenode,hmaster	2G
Hadoop2	192.168.0.11	Namenode,hiverserver2,hivemetastore	1G
Hadoop3	192.168.0.12	Zookeeper,datenode,hregionserver,journalnode	2G
Hadoop4	192.168.0.13	Zookeeper,datenode,hregionserver,journalnode	2G
Hadoop5	192.168.0.14	Zookeeper,datenode,hregionserver,journalnode	2G

2. Modify/etc/hosts
Performed on all machines

echo "127.0.0.1     localhost localhost">/etc/hostsecho "192.168.0.10  hadoop1">>/etc/hostsecho "192.168.0.11  hadoop2">>/etc/hostsecho "192.168.0.12  hadoop3">>/etc/hostsecho "192.168.0.13  hadoop4">>/etc/hostsecho "192.168.0.14  hadoop5">>/etc/hosts

3. Configure SSH Trust
It's best to have root hdfs yarn with SSH Trust
1. Generating the public key
All node execution

ssh-keygen -t rsa -P ‘‘

2. All node execution

ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.0.10ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.0.11ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.0.12ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.0.13ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.0.14

So that there is trust between all the nodes.

3. Create users and Groups
Create HDFs yarn Hive Zookeeper hbase users on all nodes and subsequently add other users

groupadd hadoop useradd -g hadoop hdfspasswd hdfs <<EOFhdfshdfsEOFuseradd -g hadoop yarnpasswd yarn <<EOFyarnyarnEOFgroupadd zookeeper useradd -g zookeeper zookeeperpasswd zookeeper <<EOFzookeeperzookeeperEOFgroupadd hive8useradd -g hive hivepasswd hive <<EOFhivehiveEOFgroupadd hbase useradd -g hbase hbasepasswd hbase <<EOFhbasehbaseEOF

4. Modifying system parameters

echo "Session required/lib64/security/pam_limits.so" >>/etc/pam.d/loginecho "@hdfs- Nofile 4096 ">>/etc/security/limits.confecho" @hive-nofile 4096 ">>/etc/security/limits.confecho" @yarn-nofile 4096 ">>/etc/security/limits.              Confecho "@zookeeper-nofile 4096" >>/etc/security/limits.confecho "@hbase- Nofile 4096 ">>/etc/security/limits.confecho" @hdfs-nproc 102 4 ">>/etc/security/limits.confecho" @hive-nproc ">>/etc/security/limit               S.confecho "@yarn-nproc" >>/etc/security/limits.confecho "@zookeeper- Nproc 1024x768 ">>/etc/security/limits.confecho" @hbase-nproc 1 024 ">>/etc/securiTy/limits.confecho "@hdfs-memlock 32768" >>/etc/security/limits.conf 

After Setup is complete, the application needs to be restarted.

5. Turn off the firewall and SELinux

#关闭防火墙service iptables stop chkconfig iptables offsetenforce 0 #关闭selinux

Permanently close selinux:

vi /etc/selinux/configSELINUX=disabled

6. Installing NTP

yum -y install ntp

On the NTP server node:

vi /etc/ntp.conf#本子网内主机都可以同步restrict 192.168.0.0 mask 255.255.0.0 nomodify#优先时间服务器server  192.168.0.10 prefer#当ntp server不可用,把本地时间做为ntp时间给端用fudge 127.127.1.0 stratum 8 #日志文件位置logfile /var/log/ntp.log

Start the NTPD service

service ntpd start

Using 192.168.0.10 as a time server, it is recommended to use the online time server more accurately.

Client node:

vi /etc/ntp.conf#优先时间服务器server  192.168.0.10 prefer#日志文件位置logfile /var/log/ntp.log

The client synchronizes the time manually first:

nptdate hadoop1

Then start the NPTD service:
Service NTPD Start

View NPT Status:
Watch Ntpq-p

7. Disable IPV6
IPV6 can cause some problems, some programs search IPV6 by default.
Reference: https://linux.cn/article-5417-1.html

vi /etc/sysctl.conf# 禁用整个系统所有接口的IPv6net.ipv6.conf.all.disable_ipv6 = 1# 禁用某一个指定接口的IPv6(例如：eth0, lo)net.ipv6.conf.lo.disable_ipv6 = 1net.ipv6.conf.eth0.disable_ipv6 = 1#使用修改生效:$ sudo sysctl -p /etc/sysctl.conf

8. Configure Environment variables
Download java1.8, unzip to/opt/. Vi/etc/profile

export JAVA_HOME=/opt/jdk1.8.0_131PATH=$JAVA_HOME/bin:$PATHexport JAVA_HOMEexport PATH

Test Java:
Java-version

From for notes (Wiz)

1.linux Environment Configuration