hadoop-2.7.3 + hive-2.3.0 + zookeeper-3.4.8 + hbase-1.3.1 fully distributed installation configuration

The recent time to build up a bit hadoop-2.7.3 + hbase-1.3.1 + zookeeper-3.4.8 + hive-2.3.0 fully distributed platform environment, online query a lot of relevant information, installation success, deliberately recorded down for reference. first, software preparation

VMware12, hadoop-2.7.3, hbase-1.3.1, zookeeper-3.4.8, hive-2.3.0, jdk-8u65-linux-x64.tar.gz

Second, Hadoop installation configuration

1. Unzip the JDK:tar-zxvf jdk-8u65-linux-x64.tar.gz-c/home/zs/app/(pre-built folder path)

2. Unzip the Hadoop tar-zxvf hadoop-2.7.0.tar.gz-c/home/zs/app/

3. Configure environment variables: Gedit/etc/proflie Open file add:

# JDK
Export java_home=/home/zs/app/jdk1.8.0_65
Export JRE_HOME=${JAVA_HOME}/JRE
Export Classpath=.:${java_home}/lib:${jre_home}/lib: $CLASSPATH
Export Path=${java_home}/bin: $PATH

# hadoop2.7.3
Export hadoop_home=/home/zs/app/hadoop-2.7.3
Mapred-site.xml

4. Configure configuration files in Hadoop

The main configuration is four files, Core-site.xml hdfs-site.xml yarn-site.xml mapred-site.xml (copy mapred-site.x) in Hadoop2.7.3/etc/hadoop Ml.template) also open the XML file with the sudo gedit command

1) core-site.xml

<configuration>   <property>
<name>hadoop.tmp.dir</name>
<value>/ home/zs/app/hadoop-2.7.3/tmp</value>
</property>
<property>
<name> fs.defaultfs</name>
<value>hdfs://master:9000</value>//Master can also be changed to machine IP
</property& gt;
  </configuration>

2) Hdfs-site.xml

Here is how many Datanode nodes are configured, where the default is the master node is the Namenode node, slave node is the Datanode node. We have two datanode nodes here.

<configuration>
<property>
<name>dfs.replication</name>
<value>2& lt;/value>
</property>
</configuration>

3) Yarn-site.xml

<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
< value>master</value>
</property>
<property>
<name> yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property >
 </configuration>

4) Mapred-site.xml

<configuration>
<property>
<name>mapreduce.framework.name</name>
< value>yarn</value>
</property>
</configuration>

5) Configure the slaves file, open with Gedit, add:

Slave1
Slave2

Tell me what the Master Machine child node is
6) Configure the hadoop-env.sh file

Export java_home=/home/zs/app/jdk1.8.0_65

    5. Configure SSH password-free login

First install SSH via command sudoapt-get installation SSH

Next use the command ssh-keygen-t RSA cost secret key, all the way to the return

Authorize with the command cat id_rsa.pub >> Authorized_keys

After configuration, you can use SSH localhost for testing, if you do not need a password to eject the following information is configured successfully

Welcome to Ubuntu 16.04 LTS (gnu/linux 4.4.0-21-generic x86_64)

* documentation:https://help.ubuntu.com/

302 packages can be updated.
4 updates are security updates.

System Restart Required * * *
Last Login:sat Dec 3 06:16:02 from 127.0.0.1

6. Clone virtual Machine two copies, as slave node

7. Modify the host name

Use sudo gedit/etc/hostname to modify the host name and host to master. The remaining two units are slave1 and slave2, respectively.

8. Modify the Hosts

Also with sudo gedit/etc/hosts, modify the contents as follows, where IP can be viewed using the ifconfig command

192.168.71.134 Master
192.168.71.135 slave1
192.168.71.136 Slave2

Three virtual machines are to be modified

At this point, the configuration environment for Hadoop is built

Validation: Formatting the Namenode node in the master node

Hadoop Namenode-format

Then start the Hadoop cluster

start-all.sh

You can then use the JPS command to view the Java processes on each machine

Master node:

30976 Jps
29922 NameNode
30134 Secondarynamenode
30286 ResourceManager

SLAVE1 node:

2567 Jps
2346 NodeManager
2171 DataNode

Slave2 node: 2306 NodeManager
2107 DataNode
2557 Jps
three, hive installation configuration (MySQL-based local mode installation) 1. Extract hive tar-zxvf apache-hive-2.3.0-bin.tar.gz-c/home/zs/app 2. Add MySQL driver:

Download the Mysql-connector-java-x.y.z-bin.jar file and place it under the Apache-hive-2.1.1-bin/lib directory.

3. Configure environment Variables Sudo/etc/profile add:

#hive
export hive_home=/home/zs/app/hive-2.3.0
export path= $PATH: $HIVE _home/bin

4. Modify Hive-site.xml (hive-default.xml copy renamed ):

    <configuration>       <property>            <name>javax.jdo.option.ConnectionURL</name>            <value>jdbc:mysql://localhost:3306/hive? createdatabaseifnotexist=true</value>           < Description>jdbc connect string for a jdbc metastore</description>            </property>          <property>            < name>javax.jdo.option.connectiondrivername</name>             <value>com.mysql.jdbc.Driver</value>             <description>driver class name for a jdbc metastore</description>             </property>                          <property >            <name> Javax.jdo.option.connectionusername</name>