Summary of problems during Hadoop 2.2.0 + HBase compilation and Installation

1. Early Stage: Prepare the node, install the operating system, configure the host name, configure the ip addresses of each machine, configure the hosts file, configure ssh password-less mutual access, configure Intranet time synchronization, and configure the java environment. Baidu

Environment used by the author

 

 

Host Name	Alias	IP	OS	Java	Description
Ubuntu-ssd	Master. Hadoop	192.168.117.124	Ubuntu12.04 64-bit	1.7.0 _ 45 64-bit	Master
Virtual-Slave1	Slave3.hadoop	192.168.117.125	CentOS6.5 mini installation x64	1.7.0 _ 45 64-bit	Slave Virtual Machine m single processer
Physical-Slave1	Slave1.hadoop	192.168.117.120	CentOS6.5 desktop x64	1.7.0 _ 45 64-bit	Slave
Physical-Slave2	Slave2.hadoop	192.168.117.121	CentOS6.5 desktop x64	1.7.0 _ 45 64-bit	Slave
Liu-Lenovo-Product	Master. hadoop. backup	192.168.117.66	Ubuntu13.10 32-bit	1.7.0 _ 25 32-bit	Ordinary PC, I Want To Do HA, and finally only serve as ntpserver

Summary of Some Problems in the configuration process:

When it comes to file distribution, I initially used mout to mount shared files on windows servers to achieve resource distribution, but there were many unstable and interrupted problems on the day, it may be because the file server's operating system was recently updated, and later the Intranet ftp service was used, but the ftp that can be used also had a problem, so it was hard to use scp.

After summary, we recommend that you use scp + scripts as the file distribution method during hadoop construction. This method is very practical and fast, especially when there is no password-free access through ssh.

 

File distribution script

Touch authorized_keys
True> authorized_keys
Cat *. pub> authorized_keys
Scp authorized_keys hadoop@slave1.hadoop:/home/hadoop/. ssh
Scp authorized_keys hadoop@slave2.hadoop:/home/hadoop/. ssh
Scp authorized_keys hadoop@slave3.hadoop:/home/hadoop/. ssh
# Scp authorized_keys hadoop@slave1.hadoop:/home/hadoop/. ssh

This is the script used by the author to configure ssh on the master. When a new node is added, the key of the new node is manually generated and the Public key is sent to the master, add the Receiving address of the node to the script, and execute the command to update the authorized_keys file of all nodes.

 

Enter yes during the first ssh connection. This can be displayed collectively when hdfs is started, without manual mutual access. (I have a group of two ssh servers ...)

 

Similarly, for the distribution of hosts files and hadoop and related components, you can also use the same script to distribute the master hosts files to all nodes.

 

Summary of some problems with ubuntu Network Configuration

Problem Discovery: When configuring the master node, I found that the configuration tool on the UI could not take effect. Later I found the following situation:

To configure the centos network, you need to modify/etc/sysconfig/network. It takes effect after the NIC is restarted.

For ubuntu, there are two configuration files: NetworkManager and/etc/network.

The previous user of ubuntu-ssd manually modified the network configuration in/etc/network, causing all the NetManager configurations to fail. The configuration tool displayed on the interface is actually the NetworkManager UI.

 

Ubuntu uses the UI to configure the network. You need to modify/etc/NetworkManager. conf of ubuntu, restart it, and go to the Manager Policy.

We recommend that you delete NetworkManager for some long posts.

 

 

Ntp Configuration Tool

If ntpserver is an intranet device, all nodes must be synchronized with the server at regular intervals.

Use crontab.

 

Java environment Configuration

This document contains a lot of information.

For ubuntu, either/etc/environment or/home/userdir/bashrc, or/etc/profile

CentOS is generally written in/etc/profile

Suggestion: Modify the java environment variables in the/etc/profile file to facilitate maintenance.

 

It is also worth noting that the direct use of hadoop released by the Apache hadoop community is for the 32bit jdk. Due to my ignorance, I cannot find the library when hdfs is started, I have been confused about the use of buildin java for a long time and have been skeptical that the java environment is incorrect.

To use 64-bit jdk, You need to recompile the hadoop file based on the source code. Similarly, hbase

 

In addition, pay attention to the use of the updata-alternative tool (it is only convenient to switch the default java version, not required, but it is recommended)

1 2 3 Next Page