Hadoop Federation Build

Source: Internet
Author: User
Tags free ssh

Networking Description:

Hadoop1.localdomain 192.168.11.81 (Namenode)

Hadoop2.localdomain 192.168.11.82 (Namenode)

Hadoop3.localdomain 192.168.11.83 (Datanode)

Hadoop4.localdomain 192.168.11.84 (Datanode)

Hadoop5.localdomain 192.168.11.85 (Datanode)

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/56/BD/wKiom1SL1N3hM64-AALXqmx7DeA143.jpg "title=" p1.png "alt=" Wkiom1sl1n3hm64-aalxqmx7dea143.jpg "/>

First step: Uninstalling OPENJDK

Rpm-qa |grep Java

#查看已经安装的与java相关的包

Rpm-e java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64

RPM-E--nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64

Rpm-e Tzdata-java-2013g-1.el6.noarch

#################################################################

Step Two: Disable IPV6

Echo Net.ipv6.conf.all.disable_ipv6=1 >>/etc/sysctl.conf

echo "Alias net-pf-10 off" >>/etc/modprobe.d/dist.conf

echo "Alias IPv6 off" >>/etc/modprobe.d/dist.conf


Reboot

##########################################################

Step three: Unpack the Hadoop archive package

Import a compressed package into the/usr directory

Cd/usr

TAR-ZXVF hadoop--642.4.0.tar.gz


Rename directory to Hadoop

MV hadoop--642.4.0 Hadoop


Install the jdk1.7, import the installation package into the/usr directory

Cd/usr

RPM-IVH jdk-7u71-linux-x64.rpm

####################################################################

Fourth step: Setting Java&hadoop Environment variables

Vi/etc/profile

#set Java Environment

Export java_home=/usr/java/jdk1.7.0_71/

Export hadoop_prefix=/usr/hadoop/

Export classpath=.: $JAVA _home/jre/lib/rt.jar: $JAVA _home/lib/dt.jar: $JAVA _home/lib/tools.jar

Export path= $PATH: $JAVA _home/bin

#set Hadoop Environment

Export hadoop_home=/usr/hadoop/

Export path= $HADOOP _home/bin: $PATH: $HADOOP _home/sbin/


#使配置生效

Source/etc/profile

or reboot the server, the configuration will also be reloaded

################################################

Fifth step: Create the desired directory for HDFs

Mkdir-p/home/hadoop/dfs/name

Mkdir-p/home/hadoop/dfs/data

##############################

Sixth step: Java enviromnet appended to the HADOOP environment variable configuration file

echo Export java_home=/usr/java/jdk1.7.0_71/>>/usr/hadoop/etc/hadoop/hadoop-env.sh

echo Export java_home=/usr/java/jdk1.7.0_71/>>/usr/hadoop/etc/hadoop/yarn-env.sh

####################################################################################


Seventh step: Free SSH key, can only on the Namenode, because under normal circumstances, by the Namenode control Datanode

See the relevant Configuration guide

####################

Eighth step: Edit the Datanode configuration file, only on Namenode

[Email protected] hadoop]# pwd

/usr/hadoop/etc/hadoop

[Email protected] hadoop]# more slaves

Hadoop3.localdomain

Hadoop4.localdomain

Hadoop5.localdomain

########################################

[Email protected] hadoop]# more Core-site.xml

<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://hadoop1.localdomain:8020</value>

</property>

#

# <property>

# <name>fs.defaultFS</name>

# <value>viewfs://nsX</value>

# </property>

#

<property>

<name>io.file.buffer.size</name>

<value>131072</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>file:/home/hadoop/tmp</value>

</property>

<property>

<name>hadoop.proxyuser.hadoop.hosts</name>

<value>*</value>

</property>

<property>

<name>hadoop.proxyuser.hadoop.groups</name>

<value>*</value>

</property>

</configuration>

###########################################

Hdfs-site.xml


<property>

<name>dfs.namenode.name.dir</name>

<value>/home/hadoop/dfs/name</value>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>/home/hadoop/dfs/data</value>

</property>

<property>

<name>dfs.replication</name>

<value>3</value>

</property>

<property>

<name>dfs.webhdfs.enabled</name>

<value>true</value>

</property>

###############################

<property>

<name>dfs.federation.nameservices</name>

<value>ns1,ns2</value>

</property>

<property>

<name>dfs.namenode.rpc-address.ns1</name>

<value>hadoop1.localdomain:8020</value>

</property>

<property>

<name>dfs.namenode.http-address.ns1</name>

<value>hadoop1.localdomain:50070</value>

</property>

<property>

<name>dfs.namenode.secondaryhttp-address.ns1</name>

<value>hadoop1.localdomain:50080</value>

</property>

<property>

<name>dfs.namenode.rpc-address.ns2</name>

<value>hadoop2.localdomain:8020</value>

</property>

<property>

<name>dfs.namenode.http-address.ns2</name>

<value>hadoop2.localdomain:50070</value>

</property>

<property>

<name>dfs.namenode.secondaryhttp-address.ns2</name>

<value>hadoop2.localdomain:50080</value>

</property>


#################################

Mapred-site.xml


<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

<property>

<name>mapreduce.jobhistory.address</name>

<value>hadoop1.localdomain:10020</value>

</property>

<property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>hadoop1.localdomain:19888</value>

</property>


###########################

Yarn-site.xml


<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<property>

<name>yarn.resourcemanager.address</name>

<value>hadoop1.localdomain:8032</value>

</property>

<property>

<name>yarn.resourcemanager.scheduler.address</name>

<value>hadoop1.localdomain:8030</value>

</property>

<property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>hadoop1.localdomain:8031</value>

</property>

<property>

<name>yarn.resourcemanager.admin.address</name>

<value>hadoop1.localdomain:8033</value>

</property>

<property>

<name>yarn.resourcemanager.webapp.address</name>

<value>hadoop1.localdomain:8088</value>

</property>

#############################

1. On the HADOOP1, copy the Hadoop folder to the other nodes without changing the configuration

Scp/usr/hadoop hadoop2:/usr/

Scp/usr/hadoop hadoop3:/usr/

Scp/usr/hadoop hadoop4:/usr/

Scp/usr/hadoop hadoop5:/usr/


2, formatted Namenode, two namenode all to format, cut Clusterid

HDFs Namenode-format-clusterid Myhadoopcluster


Myhadoopcluster is in the form of a string


3. To delete the cache before each format Namenode


rm-rf/home/hadoop/dfs/data/*

rm-rf/home/hadoop/dfs/name/*


4.

Open

start-all.sh

Shut down

stop-all.sh


Access method:

http://hadoop1.localdomain:50070/dfsclusterhealth.jsp

http://hadoop1.localdomain:50070/dfshealth.jsp

Http://hadoop1.localdomain:50070/dfshealth.html#tab-overview

Http://hadoop1.localdomain:8088/cluster/nodes

Reference reading:

Http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-hdfs/Federation.html

http://zh.hortonworks.com/blog/an-introduction-to-hdfs-federation/


Hadoop Federation Build

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.