Essence Hadoop,hbase distributed cluster and SOLR environment building

1. Machine Preparation

1.1 Prepared 2 machines, installing the Win7 system (64-bit)
Two windows physical hosts:
192.168.131.44 administrator/(password is a space)
192.168.131.67 administrator/(password is a space)
1.2. Install VMwareWorkstation8.0.3 on each machine
VMwareWorkstation8.0.3 Serial Number: Jg2td-djl95-fzzz8-du1nh-33gh7
1.3. Create a virtual machine Centos-6.5-x86_64-bin-dvd1.iso
Personalize Linux:
Full Name:hadoop
User Name:hadoop
Password:root
You can create 5 Centos-6.5-x86_64-bin machines quickly by creating a new virtual machine in Vmwareworkstation, or by cloning (already have a virtual machine). MASTER1,MASTER2,SLAVE1,SLAVE2,SOLR respectively;
192.168.131.60 Root/root
192.168.131.86 Root/root
192.168.131.81 Root/root
192.168.131.85 Root/root
192.168.131.90 Root/root
New:

Cloning:

2. Basic Environment Configuration

To facilitate later operation of the virtual machine, it is recommended to configure the basic environment for 5 CentOS.
After entering the virtual machine, we recommend that you log in as root

2.1 Fixed Machine IP
Two different ways:
First: Modify with the Linux GUI (highly recommended)
Go to the Linux GUI-right click on the two small computers in the upper right-click Edit Connections, select the current network system eth0, click the Edit button, and select the option IPv4 Add ip:192.168.1.119 subnet Mask by clicking the Add button, Manual: 255.255.255.0 gateway: 192.168.1.1, apply

    第二种：修改配置文件方式    vim /etc/sysconfig/network-scripts/ifcfg-eth0        DEVICE="eth0"        BOOTPROTO="static"           ###        HWADDR="00:0C:29:3C:BF:E7"        IPV6INIT="yes"        NM_CONTROLLED="yes"        ONBOOT="yes"        TYPE="Ethernet"        UUID="ce22eeca-ecde-4536-8cc2-ef0dc36d4a8c"        IPADDR="192.168.1.44"       ###        NETMASK="255.255.255.0"      ###        GATEWAY="192.168.1.1"        ###

2.2 Shutting down the firewall
View firewall Status: Service iptables status
Shutting down the firewall
Service Iptables Status
Service Iptables Stop
#查看防火墙开机启动状态
Chkconfig iptables–list
#关闭防火墙开机启动
Chkconfig iptables off
2.3 Modify Hostname:
Vim/etc/sysconfig/network

    NETWORKING=yes    HOSTNAME=itcast01    ###

Five machines were modified to MASTER1,MASTER2,SLAVE1,SLAVE2,SOLR.
2.4 Configuring the Hosts file
To edit the Hosts file:
Vim/etc/hosts, add the following machine information
192.168.131.60 Master1
192.168.131.86 slave1
192.168.131.81 Master2
192.168.131.85 Slave2
192.168.131.90 SOLR
Access to the Windows system for HDFS, including development-used machines, requires that IP address information be added to the Hosts file.
2.5 Configuring SSH Password-free logins between MASTER1,MASTER2,SLAVE1,SLAVE2,SOLR machines:
Generate SSH Free Login key
CD ~, go to my home directory
CD. ssh/

Copy authorized_keys files to other machine commands
SCP Authorized_keys [Email protected]:~/.ssh/
Multi-machine set SSH password-free login technique method:
1. Install SSH. sudo apt-get install SSH. When the installation is complete, a hidden folder is generated in the ~ directory (the current user home directory, which is/home/xuhui here). SSH (Ls-a can view hidden files). If you do not have this file, you can create a new one (mkdir. ssh).
2. Enter the. SSH directory below, execute on each machine: ssh-keygen-t RSA followed by a return, generating a key;
3. After completing the second step, two files are generated:
Id-rsa #私钥
Id-rsa.pub #公钥
4. In the directory of the first machine. Execute commands under SSH, cat id-rsa.pub >> Authorized_keys; After that, the Authorized_keys file appears under SSH.
5. Then copy the Authorized_keys file under the. SSH directory of the first machine to the. SSH directory of the second computer, such as: SCP Authorized_keys [Email protected]:~/.ssh/
6. Next to the. SSH directory of the second machine, you will find the file-authorized_keys just transferred, and then execute the command to add the second computer's public key, such as: Cat Id-rsa.pub >> Authorized_keys.
7. Transfer the newly generated authorized_keys of the second computer to the third computer and add the public key-id-rsa.pub of the third computer to the Authorized_keys passed from the second computer.
8. And so on, until the last computer in the cluster.
9. After the last computer in the cluster finishes adding, the generated Authorized_keys file contains the public key of all the computers in the cluster and can be added directly to the file-authorized_keys if there are additional machines in the cluster. Finally, the last generated Authorized_keys is copied to the. SSH directory of each computer in the cluster, overwriting the previous authorized_keys.
10. After completing the Nineth step, you can login to the other computer with password-free SSH on any computer in the cluster.
2.6 Time Synchronization
In the networked state, adjust all machines in the same time zone, such as Shanghai, the time can be automatically synchronized.

3. Installing the JDK

To install the JDK, the hadoop,hbase can be installed on a machine first, such as the Master1 machine. After the installation is complete, copy the relevant files and folders directly to other machines to complete all machine installation. Detailed steps are described below.
1. Unzip the JDK
#创建文件夹
Mkdir/usr/java
#解压
TAR-ZXVF jdk-7u55-linux-i586.tar.gz-c/usr/java/
(for jdk-6u45-linux-x64.bin such a need:
1) Execute the chmod +x command on the downloaded binary file
chmod +x Jdk-6u21-linux-x64.bin
2) Execute the binary file:
./jdk-6u21-linux-x64.bin)
2. Adding Java to an environment variable
Vim/etc/profile
#在文件最后添加
Export java_home=/usr/java/jdk1.6.0_45
Export path= PATH: Java_home/bin

    #刷新配置    source /etc/profile

4. Installing Hadoop

1. Uploading a Hadoop installation package

2. Unzip the Hadoop installation package
   Mkdir/cloud
   #解压到 the/cloud/directory
   TAR-ZXVF hadoop-2.2.0.tar.gz-c/cloud/

3. Modify configuration information: (5 files) under the/hadoop-2.2.0/etc/hadoop directory
   First one: hadoop-env.sh
   Modify in line 27
   Export java_home=/usr/java/jdk1.6.0_45
   The second one: Core-site.xml
   
   
   Fs.defaultfs
   hdfs://master1:9001
   
   
   Hadoop.tmp.dir
   File:/data/hadoopdata/tmp
   
   

The third one: Hdfs-site.xml


Dfs.namenode.secondary.http-address
master2:9001


Dfs.namenode.name.dir
File:/data/hadoop-2.2.0/name
True


Dfs.datanode.data.dir
File:/data/hadoop-2.2.0/data01
True


Dfs.replication
2


Dfs.permissions
False

Fourth: Mapred-site.xml.template needs to be renamed: MV Mapred-site.xml.template Mapred-site.xml


Mapreduce.framework.name
Yarn


Mapred.system.dir
File:/data/mapred/system
True


Mapred.local.dir
File:/data/mapred/local
True

Fifth one: Yarn-site.xml

Yarn.resourcemanager.address
master1:8032

Yarn.resourcemanager.scheduler.address
master1:8030

Yarn.resourcemanager.resource-tracker.address
master1:8031

Yarn.resourcemanager.admin.address
master1:8033

Yarn.resourcemanager.webapp.address
master1:8088

Yarn.nodemanager.aux-services
Mapreduce_shuffle

Yarn.nodemanager.aux-services.mapreduce.shuffle.class
Org.apache.hadoop.mapred.ShuffleHandler

1. Add Hadoop to an environment variable
   Vim/etc/profile

   export JAVA_HOME=/usr/java/jdk1.6.0_45export HADOOP_HOME=/cloud/hadoop-2.2.0export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin

5. Installing HBase

1. Copy hbase-0.96.0-hadoop2-bin.tar.gz to Linux
   Unzip to the/data directory.
   Configuration:

2. Put Hadoop Hdfs-site.xml and Core-site.xml under hbase/conf.

3. Modify CONF/HBASE-ENV.SH (4 modifications, as below)
   

4. Modify the Hbase-site.xml file:


Hbase.rootdir
Hdfs://master1:9001/hbase
The directory shared by Regionservers


Hbase.master
master1:60000
The directory shared by Regionservers


hbase.cluster.distributed
True
The mode the cluster is in. Possible values are
False:standalone and pseudo-distributed setups with managed Zookeeper
true:fully-distributed with unmanaged Zookeeper Quorum (see HBASE-ENV.SH)



Hbase.zookeeper.quorum
Slave1,slave2
The directory shared by Regionservers


Hbase.zookeeper.property.dataDir
/data/hadoopdata/hbase/zookeeper
The property is from ZooKeeper ' s config zoo.cfg.
The directory where the snapshot is stored.



5. Modify Regionservers
Vim Regionservers
Slave1
Slave2

6. Complete other machine environment installation

Copy the Master1 Hadoop directory, JDK directory, hbase directory,/etc/hosts,/etc/profile to Master2, slave1, slave2 node
1. Hadoop directory:/data/hadoop-2.2.0
scp-r/data/hadoop-2.2.0 [email protected]:/data
Scp-r/data/hadoop-2.2.0 [ Email protected]:/data
Scp-r/data/hadoop-2.2.0 [Email protected]:/data
2. JDK directory:/usr/java
scp-r/usr/java [Email protected]:/usr/java
scp-r/usr/java [email protected]:/usr /java
Scp-r/usr/java [Email protected]:/usr/java
3. Copy HBase to other nodes
Scp-r/data/hbase-0.96.0-hadoop2/ Master2:/data
Scp-r/data/hbase-0.96.0-hadoop2/slave1:/data
Scp-r/data/hbase-0.96.0-hadoop2/slave2:/data

1. /etc/hosts:
   scp/etc/hosts [Email protected]:/etc/hosts
   scp/etc/hosts [Email protected]:/etc/hosts
   scp/etc/hosts [Email protected]:/etc/hosts
2. /etc/profile:
   scp/etc/profile [Email Protected]:/etc/profile
   scp/etc/profile [Email Protected]:/etc/profile
   scp/etc/profile [Email Protected]:/etc/profile

7. Start the cluster:

It only needs to be performed on the primary node, the Master1 machine.
1. Format HDFs (Namenode) to be formatted for the first time use, just operate on Master1.
CD to the Sbin directory of the Hadoop directory on the Master1 machine
#hdfs Namenode-format

1. To start HDFs:
   ./start-all.sh
   You can view: http://192.168.131.60:50070 (HDFs management interface)
   3. Start HBase,
   Start HBase and run on the master node:
   start-hbase.sh
   Attached: Start hmaster command separately:
   ./hbase-daemon.sh Start Master

2. To view the individual node service startup status:
   Master1:
   
   Master2:
   
   SLAVE1:
   
   Slave2:
   

5. Access the Administration page via the browser
http://192.168.131.60:60010 HBase Management Interface

http://192.168.131.60:8088 Mr Management interface (yarn management interface)

Management interface for http://192.168.131.60:50070 HDFs

At this point, hbase cluster environment is built!!

8. Preparing the SOLR Machine

8.1 Machine Base Environment configuration
1. Install the JDK (simple, detailed steps)
8.2 Installing Tomcat
1. Copy the apache-tomcat-6.0.37.tar.gz to the/data/tools directory of the CentOS
2. Unzip to/data/solr-tomcat
#tar-xvzf apache-tomcat-6.0.37.tar.gz-c/data/solr-tomcat
3. Modify after installing Tomcat./conf/server.xml

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Essence Hadoop,hbase distributed cluster and SOLR environment building