hadoop2.6 Distributed Environment

1. Preface

In 3 Systems centos6.5 Linux virtual machines build a distributed Hadoop environment, the Hadoop version is 2.6, and the node IP is

192.168.17.133 192.168.17.134 192.168.17.135 2. Configure Hosts file

Hosts files are configured on 3 nodes, as follows:

192.168.17.133 Master 192.168.17.134 slave1 192.168.17.135 slave2 127.0.0.1 localhost localhost.localdomain Localhost4 localhost4.localdomain4:: 1 localhost localhost.localdomain localhost6 localhost6.localdomain6 3. Installing the Java Environment

Here choose to install the jdk1.7 version, after the installation of the successful configuration of the environment variables:

Export Java_home=/usr/java/jdk1.7.0_67-cloudera export path= $JAVA _home/bin: $PATH export classpath=.: $JAVA _home/ Lib/dt.jar: $JAVA _home/lib/tools.jar 4. Turn off firewalls and SELinux

Temporarily shutdown firewall:

Service iptables Stop

Permanently shutdown firewall reboot takes effect:

chkconfig iptables off

Temporarily shutdown SELinux:

Setenforce 0

Permanently close selinux:

Edit/etc/selinux/config, set selinux=disabled

Reboot each node after configuration is complete to make configuration effective

5. Configure password-free login first to the user's home directory (CD ~), ls view files, one of which is ". SSH", the file price is the key. The key we generate will be placed in this folder later. Now execute command generation key: Ssh-keygen-t rsa-p "" (using RSA encryption method to generate the key) returns, prompts three times to enter the information, we directly return. Go to the folder CD. SSH (you can perform LS view files after entering a folder) appends the generated public key id_rsa.pub content to Authorized_keys, executing the command: Cat id_rsa.pub >> Authorized_keys Copy each node's Authorized_keys content to each other's file, and then ssh to each other to connect to 6. NTP server builds

Here, Master is the NTP server, slave1,slave2 as the client

First edit the/etc/ntp.conf configuration file on Master, showing only part of the content: # For more information about this file, see the Man pages # ntp.conf (5), NTP_ACC (5), Ntp_auth (5), Ntp_clock (5), Ntp_misc (5), Ntp_mon (5). Driftfile/var/lib/ntp/drift # Permit time synchronization with my time source, but does not # Permit the source to query or modify The service on this system. #restrict default Kod nomodify notrap nopeer noquery #restrict-6 default kod nomodify notrap nopeer noquery Restrict default Nomodify # Permit All Access over the loopback interface. This could # is tightened as a, but to does so would multiplying some of # the administrative. Restrict 127.0.0.1 restrict-6:: 1 # Hosts on the local receptacle are pager. #restrict 192.168.1.0 Mask 255.255.255.0 nomodify notrap # Use public servers project. # Please consider joining the pool (http://www.pool.ntp.org/join.html). #server 0.centos.pool.ntp.org iburst #server 1.centos.pool.ntp.org iburst #server 2.centos.pool.ntp.org iburst #serveR 3.centos.pool.ntp.org iburst Server 127.127.1.0 #local clock fudge 127.127.1.0 stratum 8 start httpd service on server master: Services NTPD start, setting up boot: Chkconfig ntpd on

At the client slave1,slave2 stop the NTPD services: Service ntpd stop, and manually test the ability to sync Master:ntpdate master, the successful words are as follows:

To add a scheduled synchronization task in Slave1,slave2, perform crontab-e to add the following:

*/1 * * * * * * */usr/sbin/ntpdate master;hwclock–w 7. Hadoop installation

The above steps are the basic environment configuration, now officially into the Hadoop environment configuration

7.1 Extract and configure environment variables

Download the installation package to the official website after extracting to/usr/hadoop-2.6.0, and then configure the environment variables, the contents are as follows:

Export hadoop_home=/usr/hadoop-2.6.0 export path= $PATH: $HADOOP _home/bin: $HADOOP _home/sbin 7.2 Configuration hadoop-env.sh, mapred-env.sh, yarn-env.sh

Modify the three files in the $hadoop_home/etc/hadoop directory and add the Java_home configuration:

Export java_home=/usr/java/jdk1.7.0_67-cloudera 7.3 configuration Slaves

Modify the slaves file according to the actual deployment, where the Slave1,slave2 two machines are clustered slave, the contents are:

Slave1 slave2 7.4 configuration core-site.xml hadoop.tmp.dir/hadoop/tmp fs.defaultfs hdfs://master:9000 7.5 Configuration Hdfs-site.xml Dfs.replication 3 Dfs.namenode.name.dir/hadoop/dfs/name dfs.datannode.data.dir/hadoop/dfs/data dfs.permissions False 7.6 Configure Mapred-site.xml Mapreduce.framework.name yarn mapreduce.jobhistory.address master:10020 Mapreduce.jobhistory.webapp.address master:19888 7.7 Configuration yarn-site.xml yarn.nodemanager.aux-services Mapreduce_ Shuffle Yarn.nodemanager.aux-services.mapreduce.shuffle.class Org.apache.hadoop.mapred.ShuffleHandler Yarn.resourcemanager.address master:8032 yarn.resourcemanager.scheduler.address master:8030 Yarn.resourcemanager.resource-tracker.address master:8031 yarn.resourcemanager.admin.address master:8033 Yarn.resourcemanager.webapp.address master:8088 7.8 format HDFs file system

Execute command Hadoop Namenode-format and see the following information to indicate success:

7.9 Start cluster $HADOOP _home/sbin/start-dfs.sh $HADOOP _home/sbin/start-yarn.sh $HADOOP _home/sbin/mr-jobhistory-daemon.sh STAP historyserver

When you are done, you can see through JPS whether the processes of each node are starting normally, or by accessing the Web interface:

http://master:50070 http://master:8088/