How to install Hadoop 2.4 in the Ubuntu 14 (64-bit) cluster environment

Source: Internet
Author: User
Tags file system ssh hdfs dfs

After the accumulation of the front, today finally realized the cluster environment to deploy Hadoop, and successfully ran the official example.

Work as follows:

Two machines:

Namenode: Internet Small, 3G memory, machine name: yp-x100e,ip:192.168.101.130.

Datanode: Virtual machine, Win7 download VMWare10 virtual UBUNTU14, virtual machine name: ph-v370,ip:192.168.101.110

Ensure that you can ping each other, according to the machine name and IP configuration of the respective machine/etc/hosts files and/etc/hostname files, my hosts configuration is as follows

127.0.0.1 localhost  
192.168.101.130 yp-x100e  
192.168.101.110 ph-v370

Installation environment, please refer to

UBUNTU14 Hadoop Development <1> Foundation Environment installation

UBUNTU14 Hadoop Development <2> compilation 64-bit Hadoop2.4

Configure various configuration information, please refer to the

Hadoop 2.4.0 a fully distributed platform to build, configure, install

Setting up a single Node Cluster.

Precautions:

A: Master-slave SSH settings, the host in the terminal input

Ssh-keygen-t Dsa-p '-F ~/.SSH/ID_DSA

Cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Copy files to from node

SCP Authorized_keys ph-v370:~/.ssh/

In the beginning, I always failed because the user is inconsistent, resulting in the inability to login ssh, and later I created a new Hadoop user, and granted permission to the Hadoop folder

SCP Authorized_keys ph-v370:~/.ssh/

In the beginning, I always failed because the user is inconsistent, resulting in the inability to login ssh, and later I created a new Hadoop user, and granted permission to the Hadoop folder

Useradd-m Hadoop  
passwd hadoop  
chown hadoop:hadoop hadoop-2.4.0

More Wonderful content: http://www.bianceng.cnhttp://www.bianceng.cn/Servers/cloud-computing/

It's OK to reuse users of Hadoop (including starting Hadoop services, preferably with this user)

B: Execute start-dfs.sh, sub-node report exception "WARNorg.apache.hadoop.hdfs.server.datanode.DataNode:Problem connecting to server:yp-x100e /192.168.101.130:9000 "

Here is a card point, you need to modify the master node of the hosts file, comment out the 127.0.1.1 this line, you can do before the comment

Netstat-an | grep 9000

You will see that the 9000 port is occupied by 127.0.1.1, so there will be an exception

C: The command to format the file system should be

HDFs Namenode-format

D:hadoop Services and yarn services need to be started separately

start-dfs.sh  
start-yarn.sh

E: Configure all the configuration files on the primary node and copy them directly from the node

F: Unlike when doing a single node example, I need to make a specific path when copying files, such as this:

Originally directly executed  
$ bin/hdfs dfs-put etc/hadoop input  
now needs to execute  
$ bin/hdfs dfs-put etc/hadoop/user/chenph/input

G: Check process use command is: PS-EF | grep ' search content ', kill process is: Kill-s 9 process number, view firewall is: sudo ufw status

H: Access to the master node to view the state of Hadoop http://YP-X100e:50070, as shown in the following figure, there is an active from the node, that is, the Ubuntu in my virtual machine

Author: csdn Blog Yueritian

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.