The recent time to build up a bit hadoop-2.7.3 + hbase-1.3.1 + zookeeper-3.4.8 + hive-2.3.0 fully distributed platform environment, online query a lot of relevant information, installation success, deliberately recorded down for reference.
first, software preparation
VMware12, hadoop-2.7.3, hbase-1.3.1, zookeeper-3.4.8, hive-2.3.0, jdk-8u65-linux-x64.tar.gz
Second, Hadoop installation configuration
1. Unzip the JDK:tar-zxvf jdk-8u65-linux-x64.tar.gz-c/home/zs/app/(pre-built folder path)
2. Unzip the Hadoop tar-zxvf hadoop-2.7.0.tar.gz-c/home/zs/app/
3. Configure environment variables: Gedit/etc/proflie Open file add:
# JDK
Export java_home=/home/zs/app/jdk1.8.0_65
Export JRE_HOME=${JAVA_HOME}/JRE
Export Classpath=.:${java_home}/lib:${jre_home}/lib: $CLASSPATH
Export Path=${java_home}/bin: $PATH
# hadoop2.7.3
Export hadoop_home=/home/zs/app/hadoop-2.7.3
Mapred-site.xml
4. Configure configuration files in Hadoop
The main configuration is four files, Core-site.xml hdfs-site.xml yarn-site.xml mapred-site.xml (copy mapred-site.x) in Hadoop2.7.3/etc/hadoop Ml.template) also open the XML file with the sudo gedit command
1) core-site.xml
<configuration> <property>
<name>hadoop.tmp.dir</name>
<value>/ home/zs/app/hadoop-2.7.3/tmp</value>
</property>
<property>
<name> fs.defaultfs</name>
<value>hdfs://master:9000</value>//Master can also be changed to machine IP
</property& gt;
</configuration>
2) Hdfs-site.xml
Here is how many Datanode nodes are configured, where the default is the master node is the Namenode node, slave node is the Datanode node. We have two datanode nodes here.
<configuration>
<property>
<name>dfs.replication</name>
<value>2& lt;/value>
</property>
</configuration>
3) Yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
< value>master</value>
</property>
<property>
<name> yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property >
</configuration>
4) Mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
< value>yarn</value>
</property>
</configuration>
5) Configure the slaves file, open with Gedit, add:
Slave1
Slave2
Tell me what the Master Machine child node is
6) Configure the hadoop-env.sh file
Export java_home=/home/zs/app/jdk1.8.0_65
5. Configure SSH password-free login
First install SSH via command sudoapt-get installation SSH
Next use the command ssh-keygen-t RSA cost secret key, all the way to the return
Authorize with the command cat id_rsa.pub >> Authorized_keys
After configuration, you can use SSH localhost for testing, if you do not need a password to eject the following information is configured successfully
Welcome to Ubuntu 16.04 LTS (gnu/linux 4.4.0-21-generic x86_64)
* documentation:https://help.ubuntu.com/
302 packages can be updated.
4 updates are security updates.
System Restart Required * * *
Last Login:sat Dec 3 06:16:02 from 127.0.0.1
6. Clone virtual Machine two copies, as slave node
7. Modify the host name
Use sudo gedit/etc/hostname to modify the host name and host to master. The remaining two units are slave1 and slave2, respectively.
8. Modify the Hosts
Also with sudo gedit/etc/hosts, modify the contents as follows, where IP can be viewed using the ifconfig command
192.168.71.134 Master
192.168.71.135 slave1
192.168.71.136 Slave2
Three virtual machines are to be modified
At this point, the configuration environment for Hadoop is built
Validation: Formatting the Namenode node in the master node
Hadoop Namenode-format
Then start the Hadoop cluster
start-all.sh
You can then use the JPS command to view the Java processes on each machine
Master node:
30976 Jps
29922 NameNode
30134 Secondarynamenode
30286 ResourceManager
SLAVE1 node:
2567 Jps
2346 NodeManager
2171 DataNode
Slave2 node: 2306 NodeManager
2107 DataNode
2557 Jps
three, hive installation configuration (MySQL-based local mode installation) 1. Extract hive tar-zxvf apache-hive-2.3.0-bin.tar.gz-c/home/zs/app 2. Add MySQL driver:
Download the Mysql-connector-java-x.y.z-bin.jar file and place it under the Apache-hive-2.1.1-bin/lib directory.
3. Configure environment Variables Sudo/etc/profile add:
#hive
export hive_home=/home/zs/app/hive-2.3.0
export path= $PATH: $HIVE _home/bin
4. Modify Hive-site.xml (hive-default.xml copy renamed ):
<configuration> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost:3306/hive? createdatabaseifnotexist=true</value> < Description>jdbc connect string for a jdbc metastore</description> </property> <property> < name>javax.jdo.option.connectiondrivername</name> <value>com.mysql.jdbc.Driver</value> <description>driver class name for a jdbc metastore</description> </property> <property > <name> Javax.jdo.option.connectionusername</name>