Hadoop, HBase, Zookeeper Environment (detailed)

A machine
192.168.0.203 hd203:hadoop Namenode & HBase Hmaster
192.168.0.204 hd204:hadoop Datanode & hbase Hregionserver & Zookeeper
192.168.0.205 hd205:hadoop Datanode & hbase Hregionserver & Zookeeper
192.168.0.206 hd206:hadoop Datanode & hbase Hregionserver & Zookeeper
192.168.0.202 hd202:hadoop Second Namenode & HBase Hmaster
A total of 5 machines (production environment needs to be zookeeper separate installation)

Two operating systems and software versions
1 Operating system: CentOS release 5.6 (Final) x64, shutdown firewall
2 Related software:
Jdk-6u24-linux-x64.bin
Hadoop-0.20.2-cdh3u0.tar.gz
Hbase-0.90.1-cdh3u0.tar.gz
Zookeeper-3.3.3-cdh3u0.tar.gz

Three installation

1 Installing JDK (all machines)
Mkdir/usr/java
MV Jdk-6u24-linux-x64.bin/usr/java
chmod 744 Jdk-6u24-linux-x64.bin
./jdk-6u24-linux-x64.bin
Ln-s jdk1.6.0_24 Default

2 Set up the cluster platform users (all machines)
Useradd Cbcloud
passwd cbcloud Password set to 123456


3 edition/etc/hosts (all machines)
192.168.0.203 hd203
192.168.0.204 hd204
192.168.0.205 hd205
192.168.0.206 hd206
192.168.0.202 hd202

4 hd203 Login Cluster all machines password-free setting (hd203)
Su-cbcloud
SSH login This machine also need to exempt the password as set below

$ ssh-keygen-t dsa-p '-F ~/.SSH/ID_DSA
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Append the Id_dsa.pub file on the hd203 to the Authorized_keys file of the Cbcloud user of all machines
Details are not detailed
SSH all machines need to confirm for the first time

5 Installing Hadoop
5.1 Establishment directory (all machines)
Mkdir/home/cbcloud/hdtmp
Mkdir/home/cbcloud/hddata
Mkdir/home/cbcloud/hdconf
Chown-r Cbcloud:cbcloud above catalogue
Tar zxvf hadoop-0.20.2-cdh3u0.tar.gz-c/home/cbcloud
Cd/home/cbcloud
MV Hadoop-0.20.2-cdh3u0 Hadoop
Chown-r Cbcloud:cbcloud hadoop/


5.2 Configure environment variable vi/etc/profile (all machines)
Export Java_home=/usr/java/default
Export classpath= $CLASSPATH: $JAVA _home/lib: $JAVA _home/jre/lib
Export path= $JAVA _home/bin: $JAVA _home/lib: $JAVA _home/jre/bin: $PATH: $HOME/bin
Export Hadoop_home=/home/cbcloud/hadoop
Export hadoop_conf_dir=/home/cbcloud/hdconf
Export path= $PATH: $HADOOP _home/bin

Take Hadoop's profile directory out of the source directory to facilitate later upgrades to Hadoop
MV Hadoop's conf directory file inside the/home/cbcloud/hdconf

5.3 Edit Hadoop configuration file Core-site.xml
Join
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://hd203:9000</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/home/cbcloud/hdtmp/dfs/namesecondary</value>
<description>determines where on the local filesystem the DFS secondary
Name node should store the temporary images to merge.
If This is a comma-delimited list of directories then the image is
Replicated in all of the directories for redundancy.
</description>
</property>
<property>
<name>fs.checkpoint.period</name>
<value>60</value>
<description>determines where on the local filesystem the DFS secondary
Name node should store the temporary images to merge.
If This is a comma-delimited list of directories then the image is
Replicated in all of the directories for redundancy.
</description>
</property>

5.4 Editor Hdfs-site.xml
Join
<property>
<name>dfs.replication</name>
<value>3</value>
</property>

<property>
<name>dfs.data.dir</name>
<value>/home/cbcloud/hddata</value>
</property>

<property>
<name>hadoop.tmp.dir</name>
<value>/home/cbcloud/hdtmp/</value>
</property>

<property>
<name>dfs.balance.bandwidthPerSec</name>
<value>10485760</value>
</property>
<property>
<name>dfs.hosts.exclude</name>
<value>/home/cbcloud/hdconf/excludes</value>
<final>true</final>
</property>

5.5 Editor Mapred-site.xml
Join
<property>
<name>mapred.job.tracker</name>
<value>hd203:9001</value>
</property>

5.6 Editor Hadoop-env.sh
Export Java_home=/usr/java/default

5.7 Edit Masters This file specifies the secondary name machine,
Join
hd202
Edit Slaves
Join
hd204
hd205
hd206

5.8 Copies of hd203 Hadoop and hdconf to all machines
# scp-r/home/cbcloud/hadoop Cbcloud@hd204:/home/cbcloud
# scp-r/home/cbcloud/hdconf Cbcloud@hd204:/home/cbcloud

When finished, format the Hadoop file system in hd203
Perform
Hadoop Namenode-format
Start
start-all.sh
View Datanode machines in a cluster
Execute JPS
5764 Jps
18142 Datanode
18290 Tasktracker
See the above results indicate that the boot is correct
Web mode
http://hd203:50070/dfshealth.jsp
Note that the local PC hosts file will also be configured
192.168.0.203 hd203
192.168.0.204 hd204
192.168.0.205 hd205
192.168.0.206 hd206
192.168.0.202 hd202
The Web can view the cluster status and job status, etc., so that the Hadoop installation is complete

6 Installation Zookeeper (hd203)
Tar zxvf zookeeper-3.3.3-cdh3u0.tar.gz-c/home/cbcloud
On the hd204-hd206.
Mkdir/home/cbcloud/zookeeperdata

Chown-r Cbcloud:cbcloud/home/cbcloud/zookeeperdata
Chown-r cbcloud:cbcloud/home/cbcloud/zookeeper-3.3.3-cdh3u0

Edit/home/cbcloud/zookeeper-3.3.3-cdh3u0/conf/zoo.cfg

# The number of milliseconds of each tick
ticktime=2000
# The number of ticks that the initial
# Synchronization phase can take
initlimit=10
# The number of ticks that can pass between
# Sending a request and getting an acknowledgement
Synclimit=5
# The directory where the snapshot is stored.
Datadir=/home/cbcloud/zookeeperdata
# The port at which the clients'll connect
clientport=2181
server.1=hd204:2888:3888
server.2=hd205:2888:3888
server.3=hd206:2888:3888

Zookeeper of SCP hd203 to hd204,hd205,hd206
# Scp-r/home/cbcloud/zookeeper-3.3.3-cdh3u0/cbcloud@hd205:/home/cbcloud/
In hd204-206 's/home/cbcloud/zookeeperdata directory touch myID file,
The content is 1,2,3 and server number consistent chown Cbcloud:cbcloud myID

Start Zookeeper, execute on hd204-206 bin directory
# zkserver.sh Start
After startup through
# zkserver.sh Status
View status note Executing this on the centos5.6 will cause an error.
Error contacting service. It is probably not running.
By viewing the script because
echo Stat | Nc-q 1 localhost
Different NC versions, no-Q parameters, change script Remove-Q 1 can
In addition, you can pass
echo Stat | NC localhost 2181来 View status

7 Installation HBase
7.1 Establishment directory (all machines)
Mkdir/home/cbcloud/hbconf
Chown-r cbcloud:cbcloud/home/cbcloud/hbconf
Tar zxvf hbase-0.90.1-cdh3u0.tar.gz-c/home/cbcloud
Cd/home/cbcloud
MV Hbase-0.90.1-cdh3u0 HBase
Chown-r Cbcloud:cbcloud hbase/

7.2 Configuring environment variables
Vi/etc/profile (All machines) append the following
Export hbase_conf_dir=/home/cbcloud/hbconf
Export Hbase_home=/home/hadoop/hbase

Take the HBase configuration file directory from the source directory, easy to upgrade later HBase
MV HBase files within the Conf directory inside the/HOME/CBCLOUD/HBCONF

7.3 Editor Hbase-env.sh
Export hbase_opts= "$HBASE _opts-xx:+heapdumponoutofmemoryerror-xx:+useconcmarksweepgc-xx:+cmsincrementalmode"
Export Java_home=/usr/java/default
Export Hbase_manages_zk=false
Export Hbase_home=/home/cbcloud/hbase
Export Hadoop_home=/home/cbcloud/hadoop

7.4 Editor Hbase-site.xml
Join
<property>
<name>hbase.rootdir</name>
<value>hdfs://hd203:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.master</name>
<value>hd203:60000</value>
</property>
<property>
<name>hbase.master.port</name>
<value>60000</value>
<description>the Port master should bind to.</description>
</property>

<property>
<name>hbase.zookeeper.quorum</name>
<value>hd204,hd205,hd206</value>
</property>

7.5 Editor Regionservers
Join
hd204
hd205
hd206

HBase of SCP hd203 to hd204-206,202
# Scp-r/home/cbcloud/hbase/cbcloud@hd204:/home/cbcloud
# Scp-r/home/cbcloud/hbconf/cbcloud@hd204:/home/cbcloud

Su-cbcloud
Start HBase
HBase Bin directory execution on hd203
$./start-hbase.sh
Start HBase's second Hmaster
Execute on 202
$./hbase-daemon.sh Start Master

View master:http://hd203:60010/master.jsp

View the Datanode server at this time JPS
5764 Jps
32091 Hregionserver
18142 Datanode
11221 Quorumpeermain
18290 Tasktracker
The above results indicate that the boot is normal

This hadoop+zookeeper+hbase installation is complete
Boot order
1.hadoop
2.zookeeper
3.hbase
4. A second Hmaster

Stop order
1. Second Hmaster, kill-9 delete
2.hbase
3.zookeeper
4.hadoop
Note that you must stop in order,
If the first stop zookeeper and then stop hbase, basic stop not down (own test results)

The following article will publish the use of the cluster

http://zhli986-yahoo-cn.iteye.com/blog/1204199

===========================================================

And the successful Hadoop-0.20.2 and above versions have been installed.

installation package Preparation

Need to install package:

Zookeeper-3.3.2.tar.gz (Stable version)

Hbase-0.20.2.tar.gz (Stable version)

Installation steps

Installing and Configuring Zookeeper

HBase starting from 0.20.0, you need to install zookeeper first. Download zookeeper-3.2.1.tar.gz (stable version) from Apache and extract it into the/home/hdfs/directory.

(1) Create a new zookeeper directory at the Namenode node and create a new myID file in the directory.

(2) in the zookeeper-3.2.1/conf directory, the copy zoo_sample.cfg is zoo.cfg. In Zoo.cfg, change DataDir to/home/hdfs/zookeeper and add all hosts to the bottom of the file:

server.1=10.192.1.1:2888:3888

server.2=10.192.1.2:2888:3888

server.3=10.192.1.3:2888:3888

(3) Use the SCP command to copy the/home/hdfs/zookeeper-3.2.1 and/home/hdfs/zookeeper of the Namenode node to the/home/hdfs directory of all remaining hosts.

(4) Refer to the configuration in Zoo.cfg and write the respective numbers in each host myID file. such as: 10.192.1.1 into 1,10.192.1.2 write 2

(5) Execute bin/zkserver.sh start on all nodes, starting separately.

Perform bin/zkcli.sh-server xxx.xxx.xxx.xxx:2181 to check whether the specified server started successfully.

Vi/etc/profile
Export Hbase_home=/hadoop/hbase
Export path= $PATH: $HBASE _home/bin

Export Hadoop_home=/hadoop/hadoop
Export path= $PATH: $HADOOP _home/bin

Installing and configuring HBase

TAR-ZXVF hbase-0.20.6.tar.gz

CD hbase-0.20.6

MV */hadoop/hbase

Download the HBase0.20.1 version and extract it into the/home/hdfs directory of the Namenode node.

Configuration Instructions

(1) The default settings for all configuration items in the system are viewed in hbase-default.xml, and if you need to modify the value of the configuration item, add the configuration entry in Hbase-site.xml.

To install HBase in distributed mode, the most basic configuration items that need to be added are as follows:

<property>

<name>hbase.rootdir</name>

<value>hdfs://hadoop2-namenode:9000/hbase</value>

<description>the directory shared by Region servers.</description>

</property>

<property>

<name>hbase.cluster.distributed</name>

<value>true</value>

<description>the mode the cluster is in. Possible values are

False:standalone and pseudo-distributed setups with managed zookeeper

true:fully-distributed with unmanaged zookeeper Quorum (= hbase-env.sh)

</description>

</property>
<property>
      <name> Hbase.zookeeper.property.datadir</name>
      <value>/hadoop/zookeeper </value>
      <description>property from zookeeper ' s config zoo.cfg.
      the directory where the snapshot is stored.
      </description>
    </property>
 < Property>
      <name>hbase.zookeeper.property.clientport</name>
      <value>2222</value>
      <description >property from zookeeper ' s config zoo.cfg.
      the port at which the clients'll connect.
      </description>
    </property>

<property>
      <name>hbase.zookeeper.quorum</name>
       <value>hadoop2-namenode,hadoop2-datanode1,hadoop2-datanode2</value>
      <description>comma separated list of servers in the zookeeper Quorum.
      For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com".
      By default this is set to localhost for local and pseudo-distributed modes
 & nbsp;    of operation. For a fully-distributed Setup, this should is set to a full
      list of zookeeper quorum SE RVers. If Hbase_manages_zk is set in hbase-env.sh
      This is the list of servers which we'll st Art/stop zookeeper on.
      </description>
    </property>

(2) Modify the Add configuration entry in conf/hbase-env.sh:
Export java_home=/usr/java/jdk1.6.0_22

Export Hbase_manages_zk=true

Export Hadoop_conf_dir=/hadoop/hadoop-config

Hadoop_classpath to set the HBase environment, and Hbase_classpath to design the environment of Hadoop, with this line can solve the translation HBase program when the run time error

and copy the ~/hadoop-0.20.1/conf/hdfs-site.xml to the ~/hbase-3.2.1/conf/directory.

(3) Add the Zookeeper profile zoo.cfg to the classpath of all hbase hosts.

(4) Add all Datanode nodes in the hadoop-0.20.1/conf/slaves in the conf/regionservers.
Hadoop2-datanode1
Hadoop2-datanode2

Start

Hadoop, zookeeper, and HBase should be started and shut down sequentially: Start hadoop-> start zookeeper cluster-> start hbase-> stop hbase-> set

Group-> stop Hadoop.

Execute bin/hbase-daemon.sh at the Namenode node and start master. Perform bin/start-hbase.sh and bin/stop-hbase.sh scripts to start and stop the HBase service.

/hadoop/hbase/bin/hbase-daemon.sh Start Master
/hadoop/hbase/bin/hbase-daemon.sh Stop Master
/hadoop/hbase/bin/start-hbase.sh
/hadoop/hbase/bin/stop-hbase.sh

/hadoop/hbase/bin/hbase Shell

This article comes from the Linux commune website (www.linuxidc.com) original link: http://www.linuxidc.com/Linux/2012-01/52223.htm

=============================================================