Install and configure hadoop, jdk, Hbase, and phoenix in the pseudo-distributed environment under Ubuntu16.04, and set up hadoophbase

Install and configure hadoop, jdk, Hbase, and phoenix in the pseudo-distributed environment under Ubuntu16.04, and set up hadoophbase

I. Preparations

Installation Package link: https://pan.baidu.com/s/1i6oNmOd password: i6nc

Environment preparation

Modify hostname:

$ Sudo vi/etc/hostname

Why

Modify the IP Address:

$ Sudo vi/etc/network/interfaces

Auto eth0

Iface eth0 inet static

Address 192.16.13.11

Netmask 255.255.255.0

Gateway 192.16.13.254

Restart the network service to take effect:

$ Sudo ifdown eth0 & sudo ifup eth0

Modify the relationship between the ip address and the Host Name:

$ Sudo vi/etc/hosts

192.16.13.11 why

 

1.1 create a hadoop user

$ Sudo useradd-m hadoop-s/bin/bash # create a hadoop user and use/bin/bash as the shell

$ Sudo passwd hadoop # set a password for the hadoop user and enter the password twice

$ Sudo adduser hadoop sudo # Add administrator permissions for hadoop users to facilitate deployment

$ Su-hadoop # Switch current user to hadoop user

$ Sudo apt-get update # update hadoop users' apt to facilitate subsequent Software Installation

1.2 Install SSH and configure SSH login without a password

$ Sudo apt-get install openssh-server # SSH client is installed in ubuntu by default. Here SSH server is installed.

$ Ssh-keygen-t rsa

$ Sudo localhost # log on to SSH. Enter yes for the first login.

$ Exit # exit the logon ssh localhost

$ Cat./id_rsa.pub>./authorized_keys # Add authorization

 

$ Ssh localhost # login without a password, you can see the following interface

 

 

Ii. Install jdk

$ Sudo tar zxvf jdk-8u92-linux-x64.tar.gz-C/usr/lib #/unzip to/usr/lib/jvm directory

$ Cd/usr/lib/jvm # enter this directory

$ Mv jdk1.8.0 _ 92 java # rename it to java

$ Vi ~ /. Bashrc # configure environment variables for JDK

Export JAVA_HOME =/usr/lib/jvm/java

Export JRE_HOME =$ {JAVA_HOME}/jre

Export CLASSPATH =. :$ {JAVA_HOME}/lib :$ {JRE_HOME}/lib

Export PATH =$ {JAVA_HOME}/bin: $ PATH

$ Source ~ /. Bashrc # make the new environment variable take effect

$ Java-version # Check whether the installation is successful. Check the java version.

 

Install hadoop

$ Sudo tar-zxvf hadoop-2.6.2.tar.gz-C/usr/local # unzip to the/usr/local directory

$ Cd/usr/local

$ Sudo mv hadoop-2.6.2 hadoop # rename hadoop

$ Sudo chown-R hadoop./hadoop # modify File Permissions

$ Vi ~ /. Bashrc

Export HADOOP_HOME =/usr/local/hadoop

Export CLASSPATH =$ ($ HADOOP_HOME/bin/hadoop classpath): $ CLASSPATH

Export HADOOP_COMMON_LIB_NATIVE_DIR = $ HADOOP_HOME/lib/native

Export PATH = $ PATH: $ HADOOP_HOME/bin: $ HADOOP_HOME/sbin

$ Source ~ /. Bashrc # make the new environment variable take effect

Hadoop can run in a pseudo-distributed manner on a single node. The Hadoop process runs in a separate Java Process. The Node acts both as NameNode and DataNode, and reads files in HDFS. The Hadoop configuration file is located in/usr/local/hadoop/etc/hadoop/, and pseudo-distributed needs to modify 2 profile core-site.xml and hdfs-site.xml. Hadoop configuration files are in xml format. Each configuration is implemented by declaring the name and value of the property.

Add the jdk path (export JAVA_HOME =/usr/lib/jvm/java
) Add to hadoop-env.sh File

Next modify the core-site.xml file:

<Configuration>

<Property>

<Name> hadoop. tmp. dir </name>

<Value> file:/usr/local/hadoop/tmp </value>

<Description> Abase for other temporary directories. </description>

</Property>

<Property>

<Name> fs. defaultFS </name>

<Value> hdfs :/// localhost: 9000 </value>

</Property>

</Configuration>

 

Next, modify the profile hdfs-site.xml.

<Configuration>

<Property>

<Name> dfs. replication </name>

<Value> 1 </value>

</Property>

<Property>

<Name> dfs. namenode. name. dir </name>

<Value> file:/usr/local/hadoop/tmp/dfs/name </value>

</Property>

<Property>

<Name> dfs. datanode. data. dir </name>

<Value> file:/usr/local/hadoop/tmp/dfs/data </value>

</Property>

</Configuration>

Modify slaves and add node name why

The way Hadoop runs is determined by the configuration file (the configuration file is read when Hadoop is running), so if you need to switch back to non-distributed mode from pseudo-distributed mode, you need to delete the configuration items in the core-site.xml. In addition, although pseudo-distributed systems only need to be configured with fs. defaultFS and dfs. replication can be run (refer to the official tutorial), but hadoop is not configured. tmp. dir parameter, the default temporary directory used is/tmp/hadoo-hadoop, and this directory may be cleared by the system during restart, so you must re-execute the format. Therefore, we have made settings and also specified dfs. namenode. name. dir and dfs. datanode. data. dir. Otherwise, an error may occur in the next step.

After the configuration is complete, format the NameNode.

 

$./Bin/hdfs namenode-format

Start the namenode and datanode processes and view the startup results

$./Sbin/start-dfs.sh

$ Jps

After the startup is complete, you can run the jps command to determine whether the startup is successful. If the startup is successful, the following processes are listed: "NameNode", "DataNode", and "SecondaryNameNode"

5. Install Hbase

$ Sudo tar-zxf hbase-1.1.2-hadoop2-bin.tar.gz-C/usr/local # unzip to usr/local directory

$ Cd/usr/local

$ Mv./hbase-1.1.2-hadoop2./hbase # rename

$ Sudo chown-R hadoop: hadoop./hbase # Modify permissions

Configure the command line environment variable/etc/profile

Export HBASE_HOME =/usr/local/hbase

Export PATH = $ HBASE_HOME/bin: $ PATH

Modify hbase configuration file/conf/hbase-env.sh

Export JAVA_HOME =/usr/lib/jvm/java

Export HBASE_MANAGES_ZK = true

Edit. xml profile conf/hbase-site.xml

<Configuration>

<Property>

<Name> hbase. rootdir </name>

<Value> hdfs: /localhost: 9000/hbase </value>

<Description> data storage location. </Description>

</Property>

<Property>

<Name> hbase. cluster. distributed </name>

<Value> true </value>

</Property>

<Property>

<Name> hbase. zookeeper. quorum </name>

<Value> localhost </value>

</Property>

<Property>

<Name> dfs. replication </name>

<Value> 1 </value>

<Description> the number of specified replicas is 1 because of pseudo-distribution. </Description>

</Property>

</Configuration>

Description 
Hbase. rootdir: path of hbase storage configured on hdfs File System
Whether the hbase. cluster. distributed configuration is distributed
Hbase. zookeeper. quorum specifies the node on which zookeeper is configured.
Dfs. replication configuration replica count
Note: The host and port number of hbase. rootdir are consistent with the host and port number of fs. default. name in the hadoop configuration file core-site.xml
Start hbase and execute the command start-hbase.sh in the bin directory
Before starting hbase, ensure that hadoop runs normally and can write files.*

6. Install phoenix

$ Sudo tar-zxf phoenix-4.7.0-HBase-1.1-bin.tar.gz-C/usr/local # unzip to usr/local directory

$ Cd/usr/local

Test the hbase-site.xml to the phoenix./bin directory.

Test the phoenix-4.7.0-HBase-1.1-server.jar package under hbase./lib.