CentOS-64bit to compile the Hadoop-2.5. source code, and distributed installation, centoshadoop

CentOS-64bit to compile the Hadoop-2.5. source code, and distributed installation, centoshadoop

Summary
CentOS7-64bit compilation Hadoop-2.5.0 and distributed Installation

Directory

1. System Environment Description

2. Preparations before installation

2.1 disable Firewall

2.2 check ssh installation. If not, install ssh

2.3 install vim

2.4 set static IP addresses

2.5 Modify host name

2.6 create a hadoop user

2.7 configure ssh keyless Logon

3. install necessary software

3.1 install JDK

##################################### [100%]

##################################### [100%]

3.2 install other required software

3.3 install ant

3.4 install findbugs

3.5 install protobuf

4. Compile hadoop source code

4.1 configure maven central repository (changed to oschina to increase access speed)

4.2 After compilation, directory/usr/hadoop-2.5.0-src/hadoop-dist/target/hadoop-2.5.0

5. Configure hadoop

5.1 basic operations

5.2 configure all slave nodes

5.3 modify hadoop-env.sh and yarn-env.sh

5.4 modify core-site.xml

5.5 modify hdfs-site.xml

5.6 modify mapred-site.xml

5.7 configure yarn-site.xml

5.8 format namenode

5.9 start hdfs

5.10 check startup status

1. System Environment Description

CentOS 7.0x64

192.168.1.7 master
192.168.1.8 slave
192.168.1.9 slave
192.168.1.10 slave

2. Preparations before installation

2.1 disable Firewall

# Systemctl status firewalld. service -- view the Firewall status # systemctl stop firewalld. service -- disable the firewall # systemctl disable firewalld. service -- permanently disable the Firewall

2.2 check ssh installation. If not, install ssh

# Systemctl status sshd. service -- View ssh status # yum install openssh-server openssh-clients

2.3 install vim

# yum -y install vim

2.4 set static IP addresses

# vim /etc/sysconfig/network-scripts/ifcfg-eno16777736

BOOTPROTO = "static"
ONBOOT = "yes"
IPADDR0 = "192.168.1.7"
PREFIX0 = "255.255.255.0"
GATEWAY0 = "192.168.1.1"
DNS1 = "61.147.37.1"
DNS2 = "101.226.4.6"

2.5 Modify host name

# vim /etc/sysconfig/network

HOSTNAME = master

# vim /etc/hosts

192.168.1.7   master192.168.1.8   slave1192.168.1.9   slave2192.168.1.10  slave3

# Hostnamectl set-hostname master (the original host modification method under CentOS7 is invalid)

2.6 create a hadoop user

# Useradd hadoop -- create a user whose username is hadoop # passwd hadoop -- set a password for hadoop

2.7 configure ssh keyless Logon

---- The following operations on the master

# Su hadoop -- switch to hadoop user $ cd ~ -- Open the user folder $ ssh-keygen-t rsa-p' -- generate a password pair,/home/hadoop /. ssh/id_rsa and/home/hadoop /. ssh/id_rsa.pub $ cat ~ /. Ssh/id_rsa.pub> ~ /. Ssh/authorized_keys -- append id_rsa.pub to the authorization key to $ chmod 600 ~ /. Ssh/authorized_keys -- modify permission $ su -- switch to root user # vim/etc/ssh/sshd_config -- modify ssh configuration file RSAAuthentication yes # enable RSA Authentication PubkeyAuthentication yes # enable public key/private key pair Authentication method AuthorizedKeysFile. ssh/authorized_keys # public key file path # su hadoop -- switch to hadoop user $ scp ~ /. Ssh/id_rsa.pub hadoop@192.168.1.8 :~ /-- Copy the public key to all Slave machines

---- The following operations on slave1

# Su hadoop -- switch to hadoop user $ mkdir ~ /. Ssh $ chmod 700 ~ /. Ssh $ cat ~ /Id_rsa.pub> ~ /. Ssh/authorized_keys -- append to the authorization file "authorized_keys" $ chmod 600 ~ /. Ssh/authorized_keys -- modify permission $ su -- switch back to root user # vim/etc/ssh/sshd_config -- modify ssh configuration file RSAAuthentication yes # enable RSA Authentication PubkeyAuthentication yes # enable public key private key pair authentication Method AuthorizedKeysFile. ssh/authorized_keys # public key file path

3. install necessary software

3.1 install JDK

# rpm -ivh jdk-7u67-linux-x64.rpm

Preparing...

##################################### [100%]

1: jdk

##################################### [100%]

Unpacking JAR files...
Rt. jar...
Jsse. jar...
Charsets. jar...
Tools. jar...
Localedata. jar...

# Vim/etc/profile export JAVA_HOME =/usr/java/jdk1.7.0 _ 67 export PATH = $ PATH: $ JAVA_HOME/bin # source profile -- modification takes effect

3.2 install other required software

# yum install maven svn ncurses-devel gcc* lzo-devel zlib-devel autoconf automake libtool cmake openssl-devel

3.3 install ant

# tar zxvf apache-ant-1.9.4-bin.tar.gz# vim /etc/profile export ANT_HOME=/usr/local/apache-ant-1.9.4 export PATH=$PATH:$ANT_HOME/bin

3.4 install findbugs

# tar zxvf findbugs-3.0.0.tar.gz# vim /etc/profile export FINDBUGS_HOME=/usr/local/findbugs-3.0.0 export PATH=$PATH:$FINDBUGS_HOME/bin

3.5 install protobuf

# Tar zxvf protobuf-2.5.0.tar.gz (must be 2.5.0 version, otherwise an error is reported when hadoop is compiled) # cd protobuf-2.5.0 #./configure -- prefix =/usr/local # make & make install

4. Compile hadoop source code

# tar zxvf hadoop-2.5.0-src.tar.gz# cd hadoop-2.5.0-src# mvn package -Pdist,native,docs -DskipTests -Dtar

4.1 configure maven central repository (changed to oschina to increase access speed)

# vim /usr/share/mavem/conf/settings.xml<mirrors>    <mirror>        <id>nexus-osc</id>        <mirrorOf>*</mirrorOf>        <name>Nexus osc</name>        <url>http://maven.oschina.net/content/groups/public/</url>    </mirror></mirrors><profiles>    <profile>    <id>jdk17</id>    <activation>        <activeByDefault>true</activeByDefault>        <jdk>1.7</jdk>    </activation>    <properties>        <maven.compiler.source>1.7</maven.compiler.source>        <maven.compiler.target>1.7</maven.compiler.target>        <maven.compiler.compilerVersion>1.7</maven.compiler.compilerVersion>    </properties>            <repositories>           <repository>                <id>nexus</id>                <name>local private nexus</name>                <url>http://maven.oschina.net/content/groups/public/</url>                <releases>                    <enabled>true</enabled>                </releases>                <snapshots>                    <enabled>false</enabled>                </snapshots>            </repository>         </repositories>        <pluginRepositories>            <pluginRepository>                <id>nexus</id>                <name>local private nexus</name>                <url>http://maven.oschina.net/content/groups/public/</url>                <releases>                    <enabled>true</enabled>                </releases>                <snapshots>                    <enabled>false</enabled>                </snapshots>            </pluginRepository>         </pluginRepositories>    </profile></profiles>

4.2 After compilation, directory/usr/hadoop-2.5.0-src/hadoop-dist/target/hadoop-2.5.0

# ./bin/hadoop versionHadoop 2.5.0Subversion Unknown -r UnknownCompiled by root on 2014-09-12T00:47ZCompiled with protoc 2.5.0From source with checksum 423dcd5a752eddd8e45ead6fd5ff9a24This command was run using /usr/hadoop-2.5.0-src/hadoop-dist/target/hadoop-2.5.0/share/hadoop/common/hadoop-common-2.5.0.jar# file lib//native/*lib//native/libhadoop.a:        current ar archivelib//native/libhadooppipes.a:   current ar archivelib//native/libhadoop.so:       symbolic link to `libhadoop.so.1.0.0'lib//native/libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x972b31264a1ce87a12cfbcc331c8355e32d0e774, not strippedlib//native/libhadooputils.a:   current ar archivelib//native/libhdfs.a:          current ar archivelib//native/libhdfs.so:         symbolic link to `libhdfs.so.0.0.0'lib//native/libhdfs.so.0.0.0:   ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x200ccf97f44d838239db3347ad5ade435b472cfa, not stripped

5. Configure hadoop

5.1 basic operations

# cp -r /usr/hadoop-2.5.0-src/hadoop-dist/target/hadoop-2.5.0 /opt/hadoop-2.5.0# chown -R hadoop:hadoop /opt/hadoop-2.5.0# vi /etc/profile export HADOOP_HOME=/opt/hadoop-2.5.0 export PATH=$PATH:$HADOOP_HOME/bin# su hadoop$ cd /opt/hadoop-2.5.0$ mkdir -p dfs/name$ mkdir -p dfs/data$ mkdir -p tmp$ cd etc/hadoop

5.2 configure all slave nodes

$ vim slavesslave1slave2slave3

5.3 modify hadoop-env.sh and yarn-env.sh

$ vim hadoop-env.sh / vim yarn-env.shexport JAVA_HOME=/usr/java/jdk1.7.0_67

5.4 modify core-site.xml

<Configuration>
<Property>
<Name> fs. defaultFS </name>
<Value> hdfs: // master: 9000 </value>
</Property>
<Property>
<Name> io. file. buffer. size </name>
<Value> 131702 </value>
</Property>
<Property>
<Name> hadoop. tmp. dir </name>
& Lt; value & gt; file:/opt/hadoop-2.5.0/tmp & lt;/value & gt;
</Property>
<Property>
<Name> hadoop. proxyuser. hadoop. hosts </name>
<Value> </value>
</Property>
<Property>
<Name> hadoop. proxyuser. hadoop. groups </name>
<Value> </value>
</Property>
</Configuration>

5.5 modify hdfs-site.xml

<Configuration>
<Property>
<Name> dfs. namenode. name. dir </name>
<Values>/opt/hadoop-2.5.0/dfs/name </value>
</Property>
<Property>
<Name> dfs. datanode. data. dir </name>
<Value>/opt/hadoop-2.5.0/dfs/data </value>
</Property>
<Property>
<Name> dfs. replication </name>
<Value> 3 </value>
</Property>
<Property>
<Name> dfs. namenode. secondary. http-address </name>
<Value> master: 9001. </value>
</Property>
<Property>
<Name> dfs. webhdfs. enabled </name>
<Value> true </value>
</Property>
</Configuration>

5.6 modify mapred-site.xml

# cp mapred-site.xml.template mapred-site.xml

<Configuration>
<Property>
<Name> mapreduce. framework. name </name>
<Value> yarn </value>
</Property>
<Property>
<Name> mapreduce. jobhistory. address </name>
<Value> master: 10020. </value>
</Property>
<Property>
<Name> mapreduce. jobhistory. webapp. address </name>
<Value> master: 19888. </value>
</Property>
</Configuration>

5.7 configure yarn-site.xml

<Configuration>
<Property>
<Name> yarn. nodemanager. aux-services </name>
<Value> mapreduce_shuffle </value>
</Property>
<Property>
<Name> yarn. nodemanager. auxservices. mapreduce. shuffle. class </name>
<Value> org. apache. hadoop. mapred. ShuffleHandler </value>
</Property>
<Property>
<Name> yarn. resourcemanager. address </name>
<Value> master: 8032. </value>
</Property>
<Property>
<Name> yarn. resourcemanager. schedager. address </name>
<Value> master: 8030. </value>
</Property>
<Property>
<Name> yarn. resourcemanager. resource-tracker.address </name>
<Value> master: 8031. </value>
</Property>
<Property>
<Name> yarn. resourcemanager. admin. address </name>
<Value> master: 8033. </value>
</Property>
<Property>
<Name> yarn. resourcemanager. webapp. address </name>
<Value> master: 8088. </value>
</Property>
<Property>
<Name> yarn. nodemanager. resource. memory-mb </name>
<Value> 768 </value>
</Property>
</Configuration>

5.8 format namenode

$ ./bin/hdfs namenode -format

5.9 start hdfs

$ ./sbin/start-dfs.sh$ ./sbin/start-yarn.sh

5.10 check startup status

http://192.168.1.7:8088http://192.168.1.7:50070

Reference

Http://www.it165.net/admin/html/201403/2453.html

Http://blog.csdn.net/xocoder/article/details/9155901