Hadoop fully distributed configuration (2 nodes)

Hadoop fully Distributed configurationRequired Documents: jdk-8u65-linux-x64.tar.gz hadoop-2.6.0.tar.gz

Node type	IP Address	Host Name
Namenode	192.168.29.6	Master	Namenode/senddarynamenode/resourcemanager/jps
DataNode	192.168.29.7	Slave1	Datenode/nodemanager/jps
DataNode	192.168.29.8	Slave2	Datenode/nodemanager/jps

1. Turn off the firewall

Users of three virtual machines should be consistent

Shut down the three virtual machine's firewall: sudo service iptables stop 2. Three virtual machine JDK

2.1 In the root directory of Hadoop users, JDK decompression, (Hadoop user action)

TAR-ZXVF Jdk-8u65-linux-x64.tar.gz–c app/

After the decompression is complete, there is a jdk1.8.0_65 directory in the Hadoop user's app directory

2.2 Configuring environment variables, need to modify/etc/profile file (root user action)

Cut to root user, enter SU command

Vi/etc/profile

After entering the editor, enter I, enter the VI Editor's insert mode, add at the end of the profile file

Java_home=/home/hadoop/app/jdk1.8.0_65 Export path= $PATH: $JAVA _home/bin

When the edit is complete, press ESC to exit the Insert mode input: A colon is identified in the lower-left corner

Q Exit does not save

Wq Save exit

Q! Force exit

1. Take the modified environment variable into effect (Hadoop user action)

Execute Source/etc/profile 3. Configure the Hosts file for three machines

Vi/etc/hosts

Content:

192.168.29.6 Master

192.168.29.7 slave1

192.168.29.8 Slave2

You can modify master's hosts file and then copy it to another two machines:

Scp/etc/hosts slave1:/etc/hosts

Scp/etc/hosts slave2:/etc/hosts 4. Configure host names and IP addresses for three machines

4.1 Cloning slave1, Slave2

Configure IP

Address Netmask Gateway

master:192.168.29.6 255.255.255.0 192.168.29.2

slave1:192.168.29.7 255.255.255.0 192.168.29.2

slave2:192.168.29.8 255.255.255.0 192.168.29.2 5. Configure SSH Trust

1. Generate public and private keys

Execute Ssh-keygen at the command line, then enter, then prompt for content, nothing to write, all the way to the return

Under the Hadoop user root directory, there is an. SSH directory

Id_rsa private Key

Id_rsa.pub Public Key

Known_hosts via SSH link to this host, there will be a record here

2. Give the public key to the trusted host (native)

Enter the Ssh-copy-id host name at the command line

Ssh-copy-id Master

Ssh-copy-id slave1

Ssh-copy-id Slave2

The password for the trusted host needs to be entered during replication

3. Verify, enter in command line: SSH Trust host name

SSH Master

SSH slave1

SSH slave2

If you are not prompted to enter a password, the configuration succeeds

Master and slave1 mutually configure mutual trust

Master and Slave2 Mutual trust 6. Configure Hadoop on the master node

1. In the root directory of the Hadoop user, unzip (Hadoop user action)

TAR-ZXVF hadoop-2.6.0.tar.gz–c./app

Unzip to complete a hadoop-2.6.0 directory under the Hadoop user's app directory

2. Modify the configuration file (Hadoop user action)

VI hadoop-2.6.0/etc/hadoop/hadoop-env.sh Modification

Export java_home=/home/hadoop/app/jdk1.8.0_65

3. Modify the configuration file, add (Hadoop user action)

VI Hadoop-2.6.0/etc/hadoop/core-site.xml

<property> <name>fs.defaultFS</name> <value>hdfs://master:9000</value> </property> <property> <name>io.file.buffer.size</name> <value>131072</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/appdata/hadoop/tmp</value> <description>abasefor Other temporary directories.</description> </property>

4. Modify config file vi hadoop-2.6.0/etc/hadoop/hdfs-site.xml, add (Hadoop user action)

<property> <name>dfs.namenode.secondary.http-address</name> <value>master:9001</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>/home/hadoop/appdata/hadoop/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/home/hadoop/appdata/hadoop/dfs/data</value> </property> <property> <name>dfs.replication</name> <value>2</value> </property>

5. Modify config file vi etc/hadoop/mapred-site.xml (Hadoop user action)

This file does not need to be copied in a copy (CD hadoop-2.6.0/) directory

CP Etc/hadoop/mapred-site.xml.template Etc/hadoop/mapred-site.xml

Add to

<property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>master:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>master:19888</value> </property>

6. Modify config file vi etc/hadoop/yarn-site.xml, add (Hadoop user action)

<property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>master:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>master:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>master:8035</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>master:8033</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>master:8088</value> </property>

7. Modify config file vi etc/hadoop/slaves, add (Hadoop user action)

Slave1 Slave2

7. Replicate Hadoop to the other two nodes

Scp–r app/hadoop-2.6.0 slave1:~/app/

Scp–r app/hadoop-2.6.0 slave2:~/app/ 8. Cluster formatting and startup

Bin/hdfs Namenode–format

The format can only be manipulated once and formatted on the master node, and if it needs to be formatted again, the Appdata/hadoop directory needs to be emptied

Start Hadoop

sbin/start-all.sh

In master, Slave1, Slave2, respectively: JPS

Master appears JPS, Namenode, Senddarynamenode, ResourceManager

Slave1, Slave2 appeared datenode/nodemanager/jps

The configuration succeeds 9. Test 9.1 put a file into HDFs, if there is no error, the success

Enter the command in the Hadoop unzip directory

View folder: Bin/hadoop Fs-ls/

Create directory: Bin/hadoop fs-mkdir/input

Create file: VI input.txt Enter any content

Place local files into Hdfs:bin/hadoop fs-put input.txt/input

View the contents of a file in HDFs:: Bin/hadoop fs-cat/input/input.txt

Delete directory: Bin/hadoop fs-rm-r/input

9.2 Master Virtual machine under open master:50070

http://192.168.29.6:50070 10. Turn off Hadoop

sbin/stop-all.sh