Hadoop Federation Build

Networking Description:

Hadoop1.localdomain 192.168.11.81 (Namenode)

Hadoop2.localdomain 192.168.11.82 (Namenode)

Hadoop3.localdomain 192.168.11.83 (Datanode)

Hadoop4.localdomain 192.168.11.84 (Datanode)

Hadoop5.localdomain 192.168.11.85 (Datanode)

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/56/BD/wKiom1SL1N3hM64-AALXqmx7DeA143.jpg "title=" p1.png "alt=" Wkiom1sl1n3hm64-aalxqmx7dea143.jpg "/>

First step: Uninstalling OPENJDK

Rpm-qa |grep Java

#查看已经安装的与java相关的包

Rpm-e java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64

RPM-E--nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64

Rpm-e Tzdata-java-2013g-1.el6.noarch

#################################################################

Step Two: Disable IPV6

Echo Net.ipv6.conf.all.disable_ipv6=1 >>/etc/sysctl.conf

echo "Alias net-pf-10 off" >>/etc/modprobe.d/dist.conf

echo "Alias IPv6 off" >>/etc/modprobe.d/dist.conf

Reboot

##########################################################

Step three: Unpack the Hadoop archive package

Import a compressed package into the/usr directory

Cd/usr

TAR-ZXVF hadoop--642.4.0.tar.gz

Rename directory to Hadoop

MV hadoop--642.4.0 Hadoop

Install the jdk1.7, import the installation package into the/usr directory

Cd/usr

RPM-IVH jdk-7u71-linux-x64.rpm

####################################################################

Fourth step: Setting Java&hadoop Environment variables

Vi/etc/profile

#set Java Environment

Export java_home=/usr/java/jdk1.7.0_71/

Export hadoop_prefix=/usr/hadoop/

Export classpath=.: $JAVA _home/jre/lib/rt.jar: $JAVA _home/lib/dt.jar: $JAVA _home/lib/tools.jar

Export path= $PATH: $JAVA _home/bin

#set Hadoop Environment

Export hadoop_home=/usr/hadoop/

Export path= $HADOOP _home/bin: $PATH: $HADOOP _home/sbin/

#使配置生效

Source/etc/profile

or reboot the server, the configuration will also be reloaded

################################################

Fifth step: Create the desired directory for HDFs

Mkdir-p/home/hadoop/dfs/name

Mkdir-p/home/hadoop/dfs/data

##############################

Sixth step: Java enviromnet appended to the HADOOP environment variable configuration file

echo Export java_home=/usr/java/jdk1.7.0_71/>>/usr/hadoop/etc/hadoop/hadoop-env.sh

echo Export java_home=/usr/java/jdk1.7.0_71/>>/usr/hadoop/etc/hadoop/yarn-env.sh

####################################################################################

Seventh step: Free SSH key, can only on the Namenode, because under normal circumstances, by the Namenode control Datanode

See the relevant Configuration guide

####################

Eighth step: Edit the Datanode configuration file, only on Namenode

[Email protected] hadoop]# pwd

/usr/hadoop/etc/hadoop

[Email protected] hadoop]# more slaves

Hadoop3.localdomain

Hadoop4.localdomain

Hadoop5.localdomain

########################################

[Email protected] hadoop]# more Core-site.xml

<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://hadoop1.localdomain:8020</value>

</property>

#

# <property>

# <name>fs.defaultFS</name>

# <value>viewfs://nsX</value>

# </property>

#

<property>

<name>io.file.buffer.size</name>

<value>131072</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>file:/home/hadoop/tmp</value>

</property>

<property>

<name>hadoop.proxyuser.hadoop.hosts</name>

<value>*</value>

</property>

<property>

<name>hadoop.proxyuser.hadoop.groups</name>

<value>*</value>

</property>

</configuration>

###########################################

Hdfs-site.xml

<property>

<name>dfs.namenode.name.dir</name>

<value>/home/hadoop/dfs/name</value>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>/home/hadoop/dfs/data</value>

</property>

<property>

<name>dfs.replication</name>

<value>3</value>

</property>

<property>

<name>dfs.webhdfs.enabled</name>

<value>true</value>

</property>

###############################

<property>

<name>dfs.federation.nameservices</name>

<value>ns1,ns2</value>

</property>

<property>

<name>dfs.namenode.rpc-address.ns1</name>

<value>hadoop1.localdomain:8020</value>

</property>

<property>

<name>dfs.namenode.http-address.ns1</name>

<value>hadoop1.localdomain:50070</value>

</property>

<property>

<name>dfs.namenode.secondaryhttp-address.ns1</name>

<value>hadoop1.localdomain:50080</value>

</property>

<property>

<name>dfs.namenode.rpc-address.ns2</name>

<value>hadoop2.localdomain:8020</value>

</property>

<property>

<name>dfs.namenode.http-address.ns2</name>

<value>hadoop2.localdomain:50070</value>

</property>

<property>

<name>dfs.namenode.secondaryhttp-address.ns2</name>

<value>hadoop2.localdomain:50080</value>

</property>

#################################

Mapred-site.xml

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

<property>

<name>mapreduce.jobhistory.address</name>

<value>hadoop1.localdomain:10020</value>

</property>

<property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>hadoop1.localdomain:19888</value>

</property>

###########################

Yarn-site.xml

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<property>

<name>yarn.resourcemanager.address</name>

<value>hadoop1.localdomain:8032</value>

</property>

<property>

<name>yarn.resourcemanager.scheduler.address</name>

<value>hadoop1.localdomain:8030</value>

</property>

<property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>hadoop1.localdomain:8031</value>

</property>

<property>

<name>yarn.resourcemanager.admin.address</name>

<value>hadoop1.localdomain:8033</value>

</property>

<property>

<name>yarn.resourcemanager.webapp.address</name>

<value>hadoop1.localdomain:8088</value>

</property>

#############################

1. On the HADOOP1, copy the Hadoop folder to the other nodes without changing the configuration

Scp/usr/hadoop hadoop2:/usr/

Scp/usr/hadoop hadoop3:/usr/

Scp/usr/hadoop hadoop4:/usr/

Scp/usr/hadoop hadoop5:/usr/

2, formatted Namenode, two namenode all to format, cut Clusterid

HDFs Namenode-format-clusterid Myhadoopcluster

Myhadoopcluster is in the form of a string

3. To delete the cache before each format Namenode

rm-rf/home/hadoop/dfs/data/*

rm-rf/home/hadoop/dfs/name/*

4.

Open

start-all.sh

Shut down

stop-all.sh

Access method:

http://hadoop1.localdomain:50070/dfsclusterhealth.jsp

http://hadoop1.localdomain:50070/dfshealth.jsp

Http://hadoop1.localdomain:50070/dfshealth.html#tab-overview

Http://hadoop1.localdomain:8088/cluster/nodes

Reference reading:

Http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-hdfs/Federation.html

http://zh.hortonworks.com/blog/an-introduction-to-hdfs-federation/

Hadoop Federation Build