Hadoop 2.2 Yarn Distributed cluster configuration process

Setting up the Environment: jdk1.6,ssh Password-free communication

System: CentOS 6.3

Cluster configuration: Namenode and ResourceManager on a single server, three data nodes

Build User: YARN

Hadoop2.2 Download Address: http://www.apache.org/dyn/closer.cgi/hadoop/common/

Step One: Upload Hadoop 2.2 and unzip to/export/yarn/hadoop-2.2.0

The outer boot script is in the Sbin directory

Inside the called script in the bin directory

Native so files are in the Lib/native directory

Configuration Program files are placed in the Libexec

Configuration files are in the ETC directory, corresponding to the previous version of the Conf directory

All jar packages are under the Share/hadoop directory

Step Two: Configure environment variables

Add the following configuration to the ~/.BASHRC file:

Export java_home=/export/servers/jdk1.6.0_25/
export hadoop_dev_home=/export/yarn/hadoop-2.2.0
Export Hadoop_mapared_home=${hadoop_dev_home}
Export Hadoop_common_home=${hadoop_dev_home}
export Hadoop_hdfs_ Home=${hadoop_dev_home}
Export Yarn_home=${hadoop_dev_home}
export hadoop_conf_dir=${hadoop_dev_home}/ Etc/hadoop
export hdfs_conf_dir=${hadoop_dev_home}/etc/hadoop
export yarn_conf_dir=${hadoop_dev_home}/ Etc/hadoop
export path= $PATH: $HADOOP _dev_home/bin: $JAVA _home/bin: $HADOOP _dev_home/sbin

After the configuration is complete, execute the source ~/.BASHRC command

Step three: Core-site.xml hdfs-site.xml mapred-site.xml yarn-site.xml configuration Core-site.xml Configuration

<configuration > <property> <name>fs.defaultFS</name> <value>hdfs://master1:9101< /value> <description></description> </property> <property> <name>hadoop.tmp.dir
</name> <value>/export/yarn/hadoop-log/</value> <description>tmp Temporary directory </description> </property> <property> <name>io.compression.codecs</name> <value> Com.hadoop.compression.lzo.lzocodec,com.hadoop.compression.lzo.lzopcodec</value> <description> Compression Coding, Lzo</description> </property> <property> <name>io.compression.codec.lzo.class</are configured here name> <value>com.hadoop.compression.lzo.LzoCodec</value> <description>lzo corresponding class </ description> </property> <property> <name>io.native.lib.available</name> <value> True</value> <description> whether to enable local native libraries </description> </property> </configuration> 

Hdfs-site.xml

<configuration> <property> <name>dfs.namenode.name.dir</name> <value>/export/yarn/ hadoop-log/nd</value> </property> <property> <name>dfs.datanode.data.dir</name> < value>/export/yarn/hadoop-log/dd</value> </property> <property> <name> Dfs.namenode.http-address</name> <value>0.0.0.0:60176</value> <description>namenode http
  Address </description> </property> <property> <name>dfs.datanode.address</name> <value>0.0.0.0:60116</value> </property> <property> <name>dfs.datanode.ipc.address </name> <value>0.0.0.0:60126</value> </property> <property> <name> dfs.datanode.http.address</name> <value>0.0.0.0:60176</value> </property> <property > <name>dfs.secondary.http.address</name> <value>0.0.0.0:60196</valuE> </