Hadoop Classic Series (iii) 2.x true cluster installation

One installation JDK

#/etc/profile

Export Java_home=/application/search/java/default

Export path= $JAVA _home/bin: $JAVA _home/jre/bin: $PATH

Export classpath=.: $JAVA _home/lib/dt.jar: $JAVA _home/lib/tools.jar

# Source/etc/profile

Two no password SSH login (recommended this machine also get)

# Local execution: Generate key pair

Ssh-keygen-t Dsa-p '-F ~/.SSH/ID_DSA

# Remote Server execution: Authorize the remote machine with the public key

# Note: On the Namenode, for the Secondarynamenode also need to operate a local no key login

Cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

# After remote machine authorization, you can delete the public key

RM-RF ~/.ssh/id_dsa.pub

Three configuration Hadoop

1) Configure hadoop-env.sh

Export Java_home=/application/search/java/default

Can not match, the default address is/log, but in order to prevent other places to modify

Export Hadoop_log_dir=/application/search/hadoop/hadoop-2.1.0-beta/logs

2) Configure Conf/core-site.xml

If the Hadoop.tmp.dir parameter is not configured, the default temporary directory for the system is:/tmp/hadoo-hadoop. And this directory will be killed after each reboot, you must rerun the format to do, or else error

<configuration>

<property>

<name>hadoop.tmp.dir</name>

<value>/data0/search/hadoop/tmp</value>

<!--NOTE: Create a TMP folder in the/data0/search/hadoop directory first-->

<description>a base for other temporary directories.</description>

</property>

<!--file System Properties-->

<property>

<name>fs.defaultFS</name>

<value>hdfs://QT-H-0117:9000</value>

</property>

</configuration>

3) Configure Hdfs-site.xml

<configuration>

<property>

<name>dfs.replication</name>

<value>1</value>

<!--NOTE: Replication is the number of copies of the data, for example: set to 3, salve less than 3 will be the error, the number of copies does not need to set too much-->

</property>

<configuration>

<property>

<name>dfs.namenode.secondary.http-address</name>

<value>QT-H-0118:50090</value>

<description>the Secondary Namenode HTTP server address and port.</description>

</property>

4) Configure Yarn-site.xml

<property>

<description>the hostname of the rm.</description>

<name>yarn.resourcemanager.hostname</name>

<!--other ports and addresses refer to this host, so you must configure that Resource manager can be separated from Namenode-->

<value>QT-H-0117</value>

</property>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce.shuffle</value>

<description> This item is not configured, error Java.lang.IllegalStateException:Invalid shuffle port number-1 returned</description >

</property>

5) Configure Mapred-site.xml

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

<description>classic,yarn,local</description>

</property>

</configuration>

6) Configure Slaves:namenode and ResourceManager specific Datanode can not be configured

qt-h-0118

qt-h-0119

Four format

/bin/hdfs Namenode-format

Five start

/sbin/start-hfs.sh

/sbin/start-yarn.sh

Six verification, on the Namenode

Http://10.10.160.150:8088/resourcemanager Admin Page

http://10.10.160.150:50070/

Seven Local library installation

Operating system: Centos-6.2-x86_64-minimal.iso

Yum Install Lrzsz vim

See more highlights of this column: http://www.bianceng.cnhttp://www.bianceng.cn/webkf/tools/

Yum install make CMake zlib-devel openssl-devel gcc gcc+ gcc-c++

Protobuf Custom Install (hadoop2.1 version must use 2.5), protobuf-2.5.0.tar.gz

Compile

Recompile the source on the target 64-bit operating system

MVN Package-pdist,native-dskiptests-dtar

Replace file

${hadoop_home}/lib/native

The following 4 files, soft connections do not move

Libhadoop.a

libhadoop.so.1.0.0

Libhdfs.a

libhdfs.so.0.0.0

libhadoop.so-> libhadoop.so.1.0.0

libhdfs.so-> libhdfs.so.0.0.0

Libhadoop directory compiled in the source code ￥${hadoop_home_src}/hadoop-common-project/hadoop-common/target/native/target/usr/local/lib

Libhdfs directory compiled in the source code ￥${hadoop_home_src}/hadoop-hdfs-project/hadoop-hdfs/target/native/target/usr/local/lib

Tips

Local library installation method

GLIBC version Inconsistent

http://blog.csdn.net/jiedushi/article/details/7496327

Re-compiling

Http://www.jm47.com/g/60913.htm

New MapReduce Framework for Hadoop Yarn detailed

http://www.ibm.com/developerworks/cn/opensource/os-cn-hadoop-yarn/#_3.1_hadoop_0.23.0

"Additional Reading"

http://blog.csdn.net/bxyz1203/article/details/8087233

"Failure case, can continue to study"

Other Ubuntu installation Information

1) Installation protobuf2.5

Default Hadoop-2.1-beta version of the default no 64-bit local library, only 32-bit, so you need to compile

First install protobuf-2.5, otherwise error [ERROR] Failed to execute goal Org.apache.hadoop:hadoop-maven-plugins:2.1.0-beta:protoc ( COMPILE-PROTOC) on Project hadoop-common:org.apache.maven.plugin.mojoexecutionexception: ' Protoc--version ' did not Return a version-> [Help 1]

Ubuntu currently defaults to 2.4.1 and cannot be installed

Apt-get Install Protobuf-compiler

or compile the installation yourself

http://blog.csdn.net/xocoder/article/details/9155901

Download Protobuf

Download Address: Http://code.google.com/p/protobuf/downloads/list

2) Installation Zlib

Apt-get Install Zlib1g-dev