Hadoop+hbase+zookeeper installation configuration and matters needing attention

http://blog.csdn.net/franklysun/article/details/6443027

This article focuses on the installation, configuration, problem solving of hbase

For the installation of Hadoop and zookeeper and related issues, you can refer to:

Hadoop:http://blog.csdn.net/franklysun/archive/2011/05/13/6417984.aspx

Zookeeper:http://blog.csdn.net/franklysun/archive/2011/05/16/6424582.aspx

Test environment

The three systems are Ubuntu11.04, respectively:

Frank-1 (host Name: hostname): 192.168.0.100 Masters-----Namenode
Frank-2 (host Name: hostname): 192.168.0.102 slaves-------Datanode

Frank-3 (host Name: hostname): 192.168.0.103 slaves-------Datanode

Requirements

The following are the requirements to be followed by the installation of the HBase, which, if not done, will result in unpredictable exceptions, errors, and less data loss (and, of course, possible success).

1. JDK

It's needless to say, it must be (jdk6 above)

2. Hadoop

Must be installed and configured correctly, refer to http://blog.csdn.net/FranklySun/archive/2011/05/13/6417984.aspx (Note: version issue)

Version problem, this is a very important issue, we must pay attention to it, otherwise it will be very cup. That's why I've been doing this for hours (including other questions, of course).

This version of HBase would only run on Hadoop 0.20.x. It won't run on Hadoop 0.21.x (nor 0.22.x). HBase'll lose data unless it is running on a HDFS that has a durable sync. Currently only the Branch-0.20-append branch has this attribute[]. "No official releases have been made from" this branch "up" to "I" have to "build" your Hadoop from the tip of T His branch. Michael Noll has written a detailed blog, building a Hadoop 0.20.x version for HBase 0.90.2, on "How to" build a Hadoop fr Om Branch-0.20-append. Recommended.

The above is the narrative in the official document Http://hbase.apache.org/book/notsoquick.html, only to see Hadoop 0.20. X., so just start using the hadoop-0.20.0, after testing is not possible, hadoop-0.20.2 test success (of course, the HBase version here is 0.90.2, the other version of the situation is not clear).

3, SSH

It is also necessary for the above Hadoop installation process to be mentioned.

4, DNS

That is, the host name and IP settings for each node are mentioned in the Hadoop configuration.

If it is not a server configured in hosts, it can also be specified by Hbase.regionserver.dns.nameserver. (This is configured in Hbase-site.xml)

5, NTP

Time calibration for all nodes. (There is a lot of information on the Internet, omitted here)

6, Ulimit and Nproc (all nodes must be set)

The default size is 1024, with Ubuntu as an example:

Add the following configuration to the/etc/security/limits.conf file:

Hadoop-nofile 32768

Hadoop hard Nproc 32000

Hadoop Soft Nproc 32000

Add the following configuration to the/etc/pam.d/common-session file, otherwise the above configuration does not function:

Session Required Pam_limits.so

7, add the configuration in the/home/frank/hadoopinstall/hadoop-config/hdfs-site.xml file:

<property>

<name>dfs.datanode.max.xcievers</name>

<value>4096</value>

</property>

8, zookeeper (using a separate zookeeper, not hbase from the zookeeper)

Zookeeper installation configuration can refer to the http://blog.csdn.net/FranklySun/archive/2011/05/16/6424582.aspx

Where the ZOO.CFG is configured as follows:

# The number of milliseconds of each tick

ticktime=2000

# The number of ticks that the initial

# Synchronization phase can take

initlimit=10

# The number of ticks that can pass between

# Sending a request and getting an acknowledgement

Synclimit=5

# The directory where the snapshot is stored.

Datadir=/home/frank/zookeeperinstall/data

# The port at which the clients'll connect

clientport=2222

server.1=192.168.0.100:2888:3888

server.2=192.168.0.102:2888:3888

server.3=192.168.0.103:2888:3888

Installation

Very simple, download hbase-0.20.2.tar.gz, extract to the preset directory. My directory:/home/frank/hbaseinstall/

Create a link after the decompression is complete (benefits are not much said): $ ln-s hbase-0.20.2 hbase

Configuration (The following is configured according to the cluster, stand-alone configuration can refer to online data)

1. Modify Conf/hbase-env.sh

Export JAVA_HOME=/HOME/FRANK/JAVAINSTALL/JDK

Export hbase_classpath=/home/frank/hadoopinstall/hadoop/conf

Export Hbase_manages_zk=false

2, modify Hbase-site.xml, add the following content

<property>

<name>hbase.rootdir</name>

<value>hdfs://frank-1:9000/hbase</value>

</property>

<property>

<name>hbase.cluster.distributed</name>

<value>true</value>

</property>

<property>

<name>hbase.zookeeper.property.clientPort</name>

<value>2222</value>

<description>property from zookeeper ' s config zoo.cfg.

The port at which the clients'll connect.

</description>

</property>

which

Hbase.rootdir: This part of "hdfs://frank-1:9000" must be consistent with the fs.default.name in Hadoop-config/core-site.xml;

Hbase.zookeeper.property.clientPort, is because I modified the default clientport in Zoo.cfg, so here to set, the default is 2181

3, copy the/home/frank/hadoopinstall/hadoop-config/hdfs-site.xml file to the HBase folder under the Conf

4, copy the ${zookeeper_home}/conf/zoo.cfg to HBase's conf folder

5, add all the Datanode nodes in the hadoop-config/conf/slaves in the conf/regionservers.

6, delete/hbase-0.90.2/lib/hadoop-core-0.20-append-r1056497.jar

Copy/hadoop-0.20.2/hadoop-0.20.0-core.jar to/hbase-0.90.2/lib/

7, finally, the configuration of the hbase-0.20.2, copy to the other node SCP

Run

The Hadoop, zookeeper, and HBase should be started and shut down sequentially: Start the hadoop-> start zookeeper cluster-> start hbase-> stop hbase-> halt zookeeper cluster-> Stop Hadoop.

Perform bin/start-hbase.sh and bin/stop-hbase.sh scripts to start and stop the HBase service.

Test

To enter the HBase shell, test:

List Lists all tables

Create a table, such as create ' test ', ' data '

Put insert data

... (The operation of HBase can refer to online materials)