CentOS7-64bit compilation Hadoop-2.5.0 and distributed Installation

1. System Environment Description

Centos 7.0x64

192.168.1.7 master
192.168.1.8 slave
192.168.1.9 slave
192.168.1.10 slave

2. Preparations before installation 2.1 disable the Firewall

# Systemctl status firewalld. Service -- view the Firewall Status # systemctl stop firewalld. Service -- disable the firewall # systemctl disable firewalld. Service -- permanently disable the Firewall

2.2 check SSH installation. If not, install SSH

# Systemctl status sshd. Service -- View SSH status # Yum install OpenSSH-server openssh-clients

2.3 install Vim

# yum -y install vim

2.4 set static IP addresses

# vim /etc/sysconfig/network-scripts/ifcfg-eno16777736

Bootproto = "static"
Onboot = "yes"
Ipaddr0 = "192.168.1.7"
Prefix0 = "255.255.255.0"
Gateway0 = "192.168.1.1"
Dns1 = "61.147.37.1"
Dns2 = "101.226.4.6"

2.5 Modify host name

# vim /etc/sysconfig/network

Hostname = Master

# vim /etc/hosts

192.168.1.7   master192.168.1.8   slave1192.168.1.9   slave2192.168.1.10  slave3

# Hostnamectl set-hostname master (the original host modification method under centos7 is invalid)

2.6 create a hadoop user

# Useradd hadoop -- create a user whose username is hadoop # passwd hadoop -- set a password for hadoop

2.7 configure SSH keyless Logon

----------- The following operations on the master

# Su hadoop -- switch to hadoop user $ Cd ~ -- Open the user folder $ ssh-keygen-t rsa-p' -- generate a password pair,/home/hadoop /. SSH/id_rsa and/home/hadoop /. SSH/id_rsa.pub $ cat ~ /. Ssh/id_rsa.pub> ~ /. Ssh/authorized_keys -- append id_rsa.pub to the authorization key to $ chmod 600 ~ /. SSH/authorized_keys -- modify permission $ su -- switch to root user # Vim/etc/ssh/sshd_config -- modify SSH configuration file rsaauthentication yes # enable RSA Authentication pubkeyauthentication yes # enable public key/private key pair Authentication method authorizedkeysfile. SSH/authorized_keys # public key file path # su hadoop -- switch to hadoop user $ SCP ~ /. Ssh/id_rsa.pub [email protected]: ~ /-- Copy the public key to all slave machines

---------- The following operations on slave1

# Su hadoop -- switch to hadoop user $ mkdir ~ /. Ssh $ chmod 700 ~ /. Ssh $ cat ~ /Id_rsa.pub> ~ /. Ssh/authorized_keys -- append to the authorization file "authorized_keys" $ chmod 600 ~ /. SSH/authorized_keys -- modify permission $ su -- switch back to root user # Vim/etc/ssh/sshd_config -- modify SSH configuration file rsaauthentication yes # enable RSA Authentication pubkeyauthentication yes # enable public key private key pair authentication Method authorizedkeysfile. SSH/authorized_keys # public key file path

3. Install the required software 3.1 install JDK

# rpm -ivh jdk-7u67-linux-x64.rpm

Preparing...

##################################### [100%]

1: JDK

##################################### [100%]

Unpacking jar files...
Rt. jar...
JSSE. jar...
Charsets. jar...
Tools. jar...
Localedata. jar...

# Vim/etc/profile export java_home =/usr/Java/jdk1.7.0 _ 67 export Path = $ path: $ java_home/bin # source profile -- modification takes effect

3.2 install other required software

# yum install maven svn ncurses-devel gcc* lzo-devel zlib-devel autoconf automake libtool cmake openssl-devel

3.3 install ant

# tar zxvf apache-ant-1.9.4-bin.tar.gz# vim /etc/profile export ANT_HOME=/usr/local/apache-ant-1.9.4 export PATH=$PATH:$ANT_HOME/bin

3.4 install findbugs

# tar zxvf findbugs-3.0.0.tar.gz# vim /etc/profile export FINDBUGS_HOME=/usr/local/findbugs-3.0.0 export PATH=$PATH:$FINDBUGS_HOME/bin

3.5 install protobuf

# Tar zxvf protobuf-2.5.0.tar.gz (must be 2.5.0 version, otherwise an error is reported when hadoop is compiled) # cd protobuf-2.5.0 #./configure -- prefix =/usr/local # Make & make install

4. Compile hadoop source code

# tar zxvf hadoop-2.5.0-src.tar.gz# cd hadoop-2.5.0-src# mvn package -Pdist,native,docs -DskipTests -Dtar

4.1 configure Maven central repository (changed to oschina to increase access speed)

# vim /usr/share/mavem/conf/settings.xml<mirrors>    <mirror>        <id>nexus-osc</id>        <mirrorOf>*</mirrorOf>        <name>Nexus osc</name>        <url>http://maven.oschina.net/content/groups/public/</url>    </mirror></mirrors><profiles>    <profile>    <id>jdk17</id>    <activation>        <activeByDefault>true</activeByDefault>        <jdk>1.7</jdk>    </activation>    <properties>        <maven.compiler.source>1.7</maven.compiler.source>        <maven.compiler.target>1.7</maven.compiler.target>        <maven.compiler.compilerVersion>1.7</maven.compiler.compilerVersion>    </properties>            <repositories>           <repository>                <id>nexus</id>                <name>local private nexus</name>                <url>http://maven.oschina.net/content/groups/public/</url>                <releases>                    <enabled>true</enabled>                </releases>                <snapshots>                    <enabled>false</enabled>                </snapshots>            </repository>         </repositories>        <pluginRepositories>            <pluginRepository>                <id>nexus</id>                <name>local private nexus</name>                <url>http://maven.oschina.net/content/groups/public/</url>                <releases>                    <enabled>true</enabled>                </releases>                <snapshots>                    <enabled>false</enabled>                </snapshots>            </pluginRepository>         </pluginRepositories>    </profile></profiles>

4.2 After compilation, directory/usr/hadoop-2.5.0-src/hadoop-Dist/target/hadoop-2.5.0

# ./bin/hadoop versionHadoop 2.5.0Subversion Unknown -r UnknownCompiled by root on 2014-09-12T00:47ZCompiled with protoc 2.5.0From source with checksum 423dcd5a752eddd8e45ead6fd5ff9a24This command was run using /usr/hadoop-2.5.0-src/hadoop-dist/target/hadoop-2.5.0/share/hadoop/common/hadoop-common-2.5.0.jar# file lib//native/*lib//native/libhadoop.a:        current ar archivelib//native/libhadooppipes.a:   current ar archivelib//native/libhadoop.so:       symbolic link to `libhadoop.so.1.0.0‘lib//native/libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x972b31264a1ce87a12cfbcc331c8355e32d0e774, not strippedlib//native/libhadooputils.a:   current ar archivelib//native/libhdfs.a:          current ar archivelib//native/libhdfs.so:         symbolic link to `libhdfs.so.0.0.0‘lib//native/libhdfs.so.0.0.0:   ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x200ccf97f44d838239db3347ad5ade435b472cfa, not stripped

5. Configure basic hadoop 5.1 operations

# cp -r /usr/hadoop-2.5.0-src/hadoop-dist/target/hadoop-2.5.0 /opt/hadoop-2.5.0# chown -R hadoop:hadoop /opt/hadoop-2.5.0# vi /etc/profile export HADOOP_HOME=/opt/hadoop-2.5.0 export PATH=$PATH:$HADOOP_HOME/bin# su hadoop$ cd /opt/hadoop-2.5.0$ mkdir -p dfs/name$ mkdir -p dfs/data$ mkdir -p tmp$ cd etc/hadoop

5.2 configure all slave nodes

$ vim slavesslave1slave2slave3

5.3 modify hadoop-env.sh and yarn-env.sh

$ vim hadoop-env.sh / vim yarn-env.shexport JAVA_HOME=/usr/java/jdk1.7.0_67

5.4 modify core-site.xml

<Configuration>
<Property>
<Name> fs. defaultfs </Name>
<Value> HDFS: // master: 9000 </value>
</Property>
<Property>
<Name> Io. file. Buffer. Size </Name>
<Value> 131702 </value>
</Property>
<Property>
<Name> hadoop. tmp. dir </Name>
& Lt; Value & gt; file:/opt/hadoop-2.5.0/tmp & lt;/value & gt;
</Property>
<Property>
<Name> hadoop. proxyuser. hadoop. hosts </Name>
<Value></Value>
</Property>
<Property>
<Name> hadoop. proxyuser. hadoop. Groups </Name>
<Value></Value>
</Property>
</Configuration>

5.5 modify hdfs-site.xml

<Configuration>
<Property>
<Name> DFS. namenode. Name. dir </Name>
<Values>/opt/hadoop-2.5.0/dfs/name </value>
</Property>
<Property>
<Name> DFS. datanode. Data. dir </Name>
<Value>/opt/hadoop-2.5.0/dfs/Data </value>
</Property>
<Property>
<Name> DFS. Replication </Name>
<Value> 3 </value>
</Property>
<Property>
<Name> DFS. namenode. Secondary. http-address </Name>
<Value> master: 9001. </value>
</Property>
<Property>
<Name> DFS. webhdfs. enabled </Name>
<Value> true </value>
</Property>
</Configuration>

5.6 modify mapred-site.xml

# cp mapred-site.xml.template mapred-site.xml

<Configuration>
<Property>
<Name> mapreduce. Framework. Name </Name>
<Value> yarn </value>
</Property>
<Property>
<Name> mapreduce. jobhistory. address </Name>
<Value> master: 10020. </value>
</Property>
<Property>
<Name> mapreduce. jobhistory. webapp. address </Name>
<Value> master: 19888. </value>
</Property>
</Configuration>

5.7 configure yarn-site.xml

<Configuration>
<Property>
<Name> yarn. nodemanager. Aux-services </Name>
<Value> mapreduce_shuffle </value>
</Property>
<Property>
<Name> yarn. nodemanager. auxservices. mapreduce. Shuffle. Class </Name>
<Value> org. Apache. hadoop. mapred. shufflehandler </value>
</Property>
<Property>
<Name> yarn. ResourceManager. address </Name>
<Value> master: 8032. </value>
</Property>
<Property>
<Name> yarn. ResourceManager. schedager. address </Name>
<Value> master: 8030. </value>
</Property>
<Property>
<Name> yarn. ResourceManager. resource-tracker.address </Name>
<Value> master: 8031. </value>
</Property>
<Property>
<Name> yarn. ResourceManager. admin. address </Name>
<Value> master: 8033. </value>
</Property>
<Property>
<Name> yarn. ResourceManager. webapp. address </Name>
<Value> master: 8088. </value>
</Property>
<Property>
<Name> yarn. nodemanager. Resource. Memory-MB </Name>
<Value> 768 </value>
</Property>
</Configuration>

5.8 format namenode

$ ./bin/hdfs namenode -format

5.9 start HDFS

$ ./sbin/start-dfs.sh$ ./sbin/start-yarn.sh

5.10 check startup status

http://192.168.1.7:8088http://192.168.1.7:50070

Reference http://www.it165.net/admin/html/201403/2453.html

Hadoop-2.2.0 4 64-bit source code compilation and distributed installation Workshop

CentOS7-64bit compilation Hadoop-2.5.0 and distributed Installation