hadoop1.0.4+hbase0.94.2+hive0.9.0 Distributed Deployment Installation

 

Because the individual is too lazy, so many maps are not uploaded, the complete deployment of the installation of the tutorial in addition to backup. Record here to prevent the document from being lost.

hadoop1.0.4+hbase0.94.2+hive0.9.0

Distributed Deployment Installation

Directory

1 Server configuration

2 Hadoop installation

3 Hbase installation

4 Hive installation

Version control information
Version	Date	Drafts and revisions	Description
1.0	2012-11-27	Yoan.liang	First draft of the message standard (internal version: +)

 

1 Server configuration

1.1 Download and install the JDK

1.2 Modify namenode and Datanode Server's respective /etc/sysconfig/network files

Modify the following:

Networking=yes

Networking_ipv6=no

Hostname=master.2.hadoop

The Scarlet Letter section is the name of the host that needs to be modified, if the file does not have a hostname entry, add it manually

1.3 in namenode and Datanode server under the respective /etc/to establish the hostname file, if the file exists is modified, the file content is the host name you need to set

1.4 Modify namenode and Datanode Server's respective /etc/hosts files

For example:

# don't remove the following line, or various programs

# that require network functionality would fail.

127.0.0.1 localhost.localdomain localhost

192.168.29.192 Slave.1.hadoop slave1

192.168.29.191 Master.hadoop Master

192.168.29.193 Slave.2.hadoop Slave2

:: 1 localhost6.localdomain6 Localhost6

The Scarlet Letter section is the host you want to add.

1.5 need to restart the server after modifying item first to second

1.6 Establish Namenode and datanode server to avoid SSH between each other

Log in with the same existing account in each server, where root is the example.

Execute the netstat–nat command to see if the SSH service is installed and started, and then execute the ssh Test command ssh localhost after opening to port 22 on the output port .

After prompted to enter Yes, a warning prompt and asked to enter a password, Enter the password, such as the normal landing that SSH installation started successfully.

return to the account root directory path , enter the SSH command ssh-keygen-t RSA appears prompt press ENTER until completed.

Enter the . SSH directory. The directory is in the root directory of the current account and is not displayed, enter the command CD directly. SSH into the SSH production key default directory, with the LS command to see the generated key pair Id_rsa,id_rsa.pub two files.

Rename id_rsa.pub to any file that does not have a duplicate name suffix of pub, where Id_rsa.pub is renamed to Master2.pub.

Use the SCP command to copy the corresponding account on each server (including this computer ). ssh directory. Execute the command on another server cat master2.pub >> authorized_keys Import the public key into the file Authorized_keys.

Execute command <ssh< span= "" > Server name after import completes, the first server connected will be prompted , enter Yes to record the server. Now Test SSH, with command <ssh< span= "" > Hostname > can login server with password-free, then set success.

2 Hadoop Installation

2.1 Download the Hadoop package file with the wget command

Download path:
Http://mirror.bjtu.edu.cn/apache/hadoop/common/hadoop-1.0.4/hadoop-1.0.4.tar.gz

2.2 Use the command to compress the package name after the download is complete .tar>

After the decompression is complete , the Hadoop folder appears in the directory, enters Hadoop and creates the TMP folder in the directory , which is used for the temporary file storage of Hadoop.

2.3 modifying /etc/profile:
Add the following: (The specific content is modified according to your installation path, this step is only for the convenience of use)

Export hadoop_home=/opt/ hadoop-1.0.4

Export path= $PATH: $HADOOP _home/bin

2.4 into the hadoop/conf directory, modify the configuration file hadoop-env.sh, add the JDK path at the end of the file .
# Set Java environment

Export java_home=/opt/jdk1.7.0

(The red Word part is the JDK path of the current server)

If the current SSH port is not the default 22, you will need to dismiss the hadoop_ssh_opts comment and modify the port

Examples are as follows:

Export hadoop_ssh_opts= "-P 2547"

Modify the configuration file core-site.xml add hadoop.tmp.dir, Fs.default.name entries in the file

Example

Hadoop.tmp.dir

/opt/hadoop/tmp

A base for other temporary directories.

Fs.default.name

hdfs://192.168.29.143:9000

Hadoop.tmp.dir is the storage location for temporary files in Hadoop, which points to the TMP folder you just created .

The fs.default.name is the path to configure HDFs, where the default is Port 9000 of Namenode.

Modify the configuration file hdfs-site.xml to add the Dfs.replication entry in the file . The item is the number of configuration data backups. such as Datanode more can be more than a few, to note that you cannot configure more than datanode number of values

Example:

Dfs.replication

2

Modify the configuration file mapred-site.xml to add the Mapred.job.tracker entry to the file that is the host that configures the Jobtracker. As per default, the Port of Namenode 9001

Example

Mapred.job.tracker

http://192.168.29.191:9001

(The red part of the example is the jobtracker host IP that needs to be modified)

Modify the configuration file Masters, where the namenode host is specified.

After the modification is complete , copy the Hadoop folder to the other machine Datanode server and modify the JDK path to the corresponding server's JDK installation path.

The Namenode server modifies the slaves file and configures all Datanode machines in the file .

Enter the bin directory start-all.sh 。hadoop项目启动。 in the Hadoop folder in the Namenode service to perform

3    Hbase 安装

3.1   进入opt目录下用命令wget 下载Hbase0.94.2，为

Http://mirror.bit.edu.cn/apache/hbase/hbase-0.94.2/hbase-0.94.2.tar.gz

3.2 Use the command to compress the package name after the download is complete .tar>

Enter the conf folder under the extracted HBase directory and modify the hbase-site.xml file.

Add hbase.rootdir,hbase.zookeeper.quorum,Hbase.zookeeper.property.dataDir, hbase.cluster.distributed,Hbase.master,dfs.support.append Six, after modification, copy the files to the Hadoop project of each server. In the Conf folder.

Example:

Hbase.rootdir

Hdfs://master.hadoop:9000/hbase

Hbase.zookeeper.quorum

Master.hadoop,slave.1.hadoop,slave.2.hadoop,slave.3.hadoop

Hbase.zookeeper.property.dataDir

/opt/hbase/zookeeper

hbase.cluster.distributed

True

Hbase.master

master.hadoop:6000

Dfs.support.append

True

(Note: Thefull HBase server is configured on the Hbase.zookeeper.quorum item and cannot use IP only with the server name configured on the hosts, separated by commas)

3.3 Edit hbase-env.sh file

Add at the end of the file

Export java_home=/opt/jdk1.7.0

Export Hbase_ssh_opts= ""

Export Hbase_pid_dir=/opt/hbase/pids

Export Hbase_manages_zk=true

3.4 Edit Modify conf/regionservers file

Master.hadoop

Slave.1.hadoop

Slave.2.hadoop

Slave.3.hadoop

The region server for HBase is added here

3.5 Copy the entire HBase folder to the same path as the server that needs to be deployed, and modify the corresponding JDK and other related paths.

3.6 Enter the Bin folder under the HBase directory and run start-hbase.sh to start the hbase service. (e.g.)

4 Hive Installation

4.1 Install Mysql and create users for hive and turn on remote login permissions

4.2 download mysql database corresponding driver jar package and put it under/home/hive-0.9.0/lib.

4.3 on the namenode server, download the hive Project with the wget command

:http://mirror.bit.edu.cn/apache/hive/hive-0.9.0/hive-0.9.0.tar.gz

Note: Thehive project only needs to be installed on the namenode server

4.4 Extracting the downloaded project compression package with the TAR command

4.5 Delete the existing Hbase-0.92.0.jar and Hbase-0.92.0-tests.jar in the Lib folder under the Hadoop project and bring the corresponding Copy the jar package.

hadoop1.0.4+hbase0.94.2+hive0.9.0 Distributed Deployment Installation