Hadoop fully distributed configuration (2 nodes)

Source: Internet
Author: User
Tags create directory ssh hadoop fs firewall
Hadoop fully Distributed configurationRequired Documents: jdk-8u65-linux-x64.tar.gz hadoop-2.6.0.tar.gz

Node type

IP Address

Host Name










1. Turn off the firewall

Users of three virtual machines should be consistent

Shut down the three virtual machine's firewall: sudo service iptables stop 2. Three virtual machine JDK

2.1 In the root directory of Hadoop users, JDK decompression, (Hadoop user action)

TAR-ZXVF Jdk-8u65-linux-x64.tar.gz–c app/

After the decompression is complete, there is a jdk1.8.0_65 directory in the Hadoop user's app directory

2.2 Configuring environment variables, need to modify/etc/profile file (root user action)

Cut to root user, enter SU command


After entering the editor, enter I, enter the VI Editor's insert mode, add at the end of the profile file


Export path= $PATH: $JAVA _home/bin

When the edit is complete, press ESC to exit the Insert mode input: A colon is identified in the lower-left corner

Q Exit does not save

Wq Save exit

Q! Force exit

1. Take the modified environment variable into effect (Hadoop user action)

Execute Source/etc/profile 3. Configure the Hosts file for three machines


Content: Master slave1 Slave2

You can modify master's hosts file and then copy it to another two machines:

Scp/etc/hosts slave1:/etc/hosts

Scp/etc/hosts slave2:/etc/hosts 4. Configure host names and IP addresses for three machines

4.1 Cloning slave1, Slave2

Configure IP

Address Netmask Gateway



slave2: 5. Configure SSH Trust

1. Generate public and private keys

Execute Ssh-keygen at the command line, then enter, then prompt for content, nothing to write, all the way to the return

Under the Hadoop user root directory, there is an. SSH directory

Id_rsa private Key

Id_rsa.pub Public Key

Known_hosts via SSH link to this host, there will be a record here

2. Give the public key to the trusted host (native)

Enter the Ssh-copy-id host name at the command line

Ssh-copy-id Master

Ssh-copy-id slave1

Ssh-copy-id Slave2

The password for the trusted host needs to be entered during replication

3. Verify, enter in command line: SSH Trust host name

SSH Master

SSH slave1

SSH slave2

If you are not prompted to enter a password, the configuration succeeds

Master and slave1 mutually configure mutual trust

Master and Slave2 Mutual trust 6. Configure Hadoop on the master node

1. In the root directory of the Hadoop user, unzip (Hadoop user action)

TAR-ZXVF hadoop-2.6.0.tar.gz–c./app

Unzip to complete a hadoop-2.6.0 directory under the Hadoop user's app directory

2. Modify the configuration file (Hadoop user action)

VI hadoop-2.6.0/etc/hadoop/hadoop-env.sh Modification

Export java_home=/home/hadoop/app/jdk1.8.0_65

3. Modify the configuration file, add (Hadoop user action)

VI Hadoop-2.6.0/etc/hadoop/core-site.xml












<description>abasefor Other temporary directories.</description>


4. Modify config file vi hadoop-2.6.0/etc/hadoop/hdfs-site.xml, add (Hadoop user action)

















5. Modify config file vi etc/hadoop/mapred-site.xml (Hadoop user action)

This file does not need to be copied in a copy (CD hadoop-2.6.0/) directory

CP Etc/hadoop/mapred-site.xml.template Etc/hadoop/mapred-site.xml

Add to













6. Modify config file vi etc/hadoop/yarn-site.xml, add (Hadoop user action)





























7. Modify config file vi etc/hadoop/slaves, add (Hadoop user action)



7. Replicate Hadoop to the other two nodes

Scp–r app/hadoop-2.6.0 slave1:~/app/

Scp–r app/hadoop-2.6.0 slave2:~/app/ 8. Cluster formatting and startup

Bin/hdfs Namenode–format

The format can only be manipulated once and formatted on the master node, and if it needs to be formatted again, the Appdata/hadoop directory needs to be emptied

Start Hadoop


In master, Slave1, Slave2, respectively: JPS

Master appears JPS, Namenode, Senddarynamenode, ResourceManager

Slave1, Slave2 appeared datenode/nodemanager/jps

The configuration succeeds 9. Test 9.1 put a file into HDFs, if there is no error, the success

Enter the command in the Hadoop unzip directory

View folder: Bin/hadoop Fs-ls/

Create directory: Bin/hadoop fs-mkdir/input

Create file: VI input.txt Enter any content

Place local files into Hdfs:bin/hadoop fs-put input.txt/input

View the contents of a file in HDFs:: Bin/hadoop fs-cat/input/input.txt

Delete directory: Bin/hadoop fs-rm-r/input

9.2 Master Virtual machine under open master:50070 10. Turn off Hadoop


Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.