Set up the various problems encountered in the Hadoop cluster, sorted as follows:

&http://www.aliyun.com/zixun/aggregation/37954.html ">nbsp; With the companion to build Hadoop cluster encountered various problems, sorted as follows:


Preface

A period of time before the winter vacation, began to investigate Hadoop2.2.0 build process, at that time suffer from no machine, just in 3 notebook, simple run through some data. In the blink of an eye two months have passed, some things have been forgotten. Now the lab has applied and divided 10 machines (4g+500g), which is enough for us to play. Start to build Hadoop2.2.0 distributed cluster, also take advantage of this opportunity to comb the whole process.

There are many blogs about the Hadoop2.2.0 version of the installation process, some very full, but do, or there will be some problems stuck there. Sometimes you need to combine several documents to build a platform. In this blog summary will be the problems we encounter, as well as some of the things that happen in the building process. For the specific installation process and configuration of the Hadoop file, we will follow up.

If you decide to take some time to read this article, please read it carefully, because each point, we have been delayed some time, comb, if you encounter the corresponding problems, but also to provide you with a solution.

1, System environment-Configure static IP:

Ubuntu Environment, here we use the 32-bit 12.04.2 version. We originally built it in old version 10.04 and there was a problem installing SSH. Later, in order to manage the cluster, we upgraded Ubuntu to 12.04 (reload Ubuntu).

In a nutshell, the Ubuntu installation process, in the Windows system, the use of Wubi installation is the simplest, click on their own installation, and then follow the steps to perform. At the end of the meeting, there was a tricky problem: The new Ubuntu couldn't surf the internet. This is a prerequisite for setting up a Hadoop environment to ensure internet access.

WORKAROUND: Configure static IP.

In the ubuntu12.04 version, the upper-right corner, there is a top and bottom icon, click Edit Connection, manually set static IP, gateway, subnet mask, DNS. The goal: to ensure that Ubuntu Internet access is the first step.

The above is a graphical configuration of static IP, we can also be manually configured through the following steps.

Implementation：

sudo gedit/etc/network/interfaces

Enter：

[plain] View plaincopyauto eth0 iface eth0 inet static address 172.16.128.136 netmask 255.255.255.0 gateway 172.16.128.1

Save: Reboot Gateway

sudo/etc/init.d/networking restart

2, install JDK

There are some problems here. The reason is that if you are new to the new Ubuntu will basically not encounter this problem, you can follow the steps below (1) can configure the environment variables, through the java-version can view JDK version. If you are operating on someone else's computer, the original JDK version is inconsistent. To reinstall the JDK, it does not affect the JDK version under someone else's user.

The solution is to extract the JDK you want to install to a local user such as/home/zz/jvm/jdk1.7.0_45, and then configure the. BASHRC environment variable. End save, source. BASHRC, java-version view JDK version number.

If we put the unpacked JDK on the desktop. US: CD desktop.

Implementation：

sudo cp-r jvm/usr/lib

Problem: In general, we have a permission problem when we transplant the JDK between different machines. This will cause even if we configure the environment variables in the normal steps, execute the following statement if you are experiencing permission problems after java-version.

Execution: JDK in JVM folder

sudo chmod-r 777 JVM pays all permissions to the JVM

The sudo chown-r zz:zz JVM pays the JVM permissions to the user, and ZZ is the current user.

2.1, JDK installation: Download path

Http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

Selection: jdk-7u45-linux-i586.tar.gz

Decompression: TAR-ZXVF jdk-7u45-linux-i586.tar.gz

Unzip the file as: jdk1.7.0_45, you can specify the path decompression or decompression after the file copied to the specified path.

To configure environment variables:

Execution: The CD enters the root directory.

sudo gedit. BASHRC

Add to:

[plain] View plaincopyexport java_home=/home/zz/jvm/jdk1.7.0_45 export JRE_HOME=/HOME/ZZ/JVM/JDK1.7.0_45/JRE Export classpath=.: $CLASSPATH: $JAVA _home/lib: $JRE _home/lib export path= $PATH: $JAVA _home/bin: $JRE _home/bin

Save, Close, execute

source. BASHRC

Java-version View Version

2.2, if there is no normal display

sudo gedit/etc/profile:

Add the contents above in. bashrc. Save, execute.

The above process applies to all processes. When you are in the company, give you the account of the virtual machine, all of your operations can not affect the performance of others. For example: you want to press chapter jdk1.7, and the server version of the JDK is 1.6, this requires that you can only extract the JDK to your user's directory, when you configure the environment variables, specify the directory you unzipped, which makes it possible to display a different version of the JDK for each user. If you are new to the new Ubuntu, the above problems may not meet you, but also missed a learning opportunity.

2.3 Overwrite the original JDK (straightforward, I do)

For cluster management, select the same installation path. Select Not in this user, extract to/usr/lib/jvm/jdk1.7.

Configuration environment variables, if in/etc/profile file, configured, after the execution of source profile still JDK version does not come out. In this current user. bashrc file, add the export path name that you just made. Close, executing source. bashrc. Java-version will appear with the JDK version.

Why we take this path: because the JDK path in the Hadoop cluster previously built in the notebook is this path, and also for compatibility.

Add separately:

[plain] View plaincopyexport java_home=/usr/lib/jvm/jdk1.7.0_45 export JRE_HOME=/USR/LIB/JVM/JDK1.7.0_45/JRE Export classpath=.: $CLASSPATH: $JAVA _home/lib: $JRE _home/lib export path= $PATH: $JAVA _home/bin: $JRE _home/bin 3, Change the Ubuntu host name

sudo gedit/etc/hostname

Add: Cluster1

Each host to perform this step, the difference is cluster1-10. Here we have 10 nodes.

Reboot, terminal display changed from original:zz@ubuntu~& to zz@cluster1-10~$

This is to install SSH later, the various hosts can be used to interconnect through SSH.

4, configure Hosts file

sudo gedit/etc/hosts

Add content, as follows: IP address and name is based on your host machine number to set.

[plain] view plaincopy127.0.0.1 localhost # The following lines are desirable for IPV6 capable hosts:: 1 ip6-localhost IP  6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 172.16.128.135  Cluster1 172.16.128.136 cluster2 172.16.128.123 cluster3 172.16.128.124 cluster4 172.16.128.134 cluster5 172.16.128.133 Cluster6 5, install SSH

After the new installation of Ubuntu.

Execute first: sudo apt update updates the latest files.

Then install Ssh:sudo apt Install openssh-server

The next thing to do is that SSH generates secret keys, and each host can ping each other.

Go to root directory execution:

SSH-KEYGEN-T RSA

CP Id_rsa.pub Authorized_keys

All machines perform the above steps: Then, gedit Authorized_keys copies the contents of the Authorized_keys from each machine to a file and copies it to each machine through the SCP. For example, if you have 6 machines, copy the contents of the Authorized_keys in the 6 Taichung to the write Authorized_keys.

For example: The following cluster1, Cluster2, Cluster3 and other content are copied to Authorized_keys, the purpose is to each host ping to each other.

[Plain] View Plaincopyssh-rsa aaaab3nzac1yc2eaaaadaqabaaabaqdntv1h/ldg5njt3+jjls6sgcidis9tq0ceslcn0lonzno/  nvavnw79mknj0lwuodv/ozz7aq0ddsbos9we8in9wqvvo2t2eoauwexu5pqcv1tsrjxj43rkfcbjjedlxt+ 4sirgqrlrwocmlosoakncislxsq2a7mxuq+njyvynyjfyykjc+p7nl0rrnhllzfy28etf3jzygkoohdidqi dA8O6xF8VsJOUTaqIc/ g0rlhuhpzgapemro+hwjhyda4uermnsalhuhbrq2pcnz0wdehjtf2psdxvihzenms+ yjgh501mjcenkwyediqhefwc9j3jegk0uazdkzbyz+ VoR zz@cluster2 Ssh-rsa aaaab3nzac1yc2eaaaadaqabaaabaqcqqbqxmsaicckcy6vwkhujvygb88ugfi/v7i407vt9mnd CeP2yRUyn+ HLZUZPXMCVQXSYDQUSWUID8FYXZI3A6UKU2B7K+7JUWZFJ8TO5L3R4NAWXN1ZQBK8SG0UBFBWC xphoa/krzq3h4tdfvhdivtdpg5chtwnlu3/  jchmldnypcocnyfndi6d/idarp/ci4rdgbv4xddor65ex47kg 7i4zxlyeajqoq9ibbsigkxrve1cfbp79dcncpelmdwkcnri3xa0rh3o5a7mliidulhqcn8kpkory55farme3 5K1bLV7rDmLdZVIY5GKdR7GgR/ 56wgzxw3czpvlfdbfdz Zz@cluster1

Finally, Authorized_keys owns the contents of all host Authorized_keys in a host.

Go to the current path and execute:

SCP Authorized_keys Cluster1:/home/zz/.ssh

And then constantly change cluster the numbers behind it, you can copy the files to different hosts.

Implementation：

SSH Cluster1 or 2 or 3, and so on. You can log on without a password.

Summary:

If, through the configuration SSH password-free login, you can through SSH cluster1 and so on between the password-free interconnection. It shows that the whole work is progressing smoothly. By this step, it also shows that our transplant work is progressing smoothly. We can then migrate the entire directory of previously built Hadoop to the current user and then configure the master and Solve nodes to start Hadoop. This is the work below.

Original link: http://blog.csdn.net/yczws1/article/details/21633433