Centos7-64bit compiled Hadoop-2.5.0, and distributed installation

Summary
Centos7-64bit compiled Hadoop-2.5.0, and distributed installation

Catalogue [-]

• 1. System Environment Description
• 2. Pre-installation preparations
• 2.1 Shutting down the firewall
• 2.2 Check SSH installation, if not, install SSH
• 2.3 Installing VIM
• 2.4 Setting a static IP address
• 2.5 Modifying the host name
• 2.6 Creating a Hadoop user
• 2.7 Configuring SSH without key login
• 3. Install the necessary software
• 3.1 Installing the JDK
• ##################################### [100%]
• ##################################### [100%]
• 3.2 Installing other required software
• 3.3 Installing Ant
• 3.4 Installing FindBugs
• 3.5 Installing PROTOBUF
• 4. Compiling Hadoop source code
• 4.1 Maven central repository configuration (changed to Oschina, increased access speed)
• 4.2 After the compilation is complete, the directory/usr/hadoop-2.5.0-src/hadoop-dist/target/hadoop-2.5.0
• 5. Configure Hadoop
• 5.1 Basic operations
• 5.2 Configuring all Slave nodes
• 5.3 Modifying hadoop-env.sh and yarn-env.sh
• 5.4 Modifying Core-site.xml
• 5.5 Modifying Hdfs-site.xml
• 5.6 Modifying Mapred-site.xml
• 5.7 Configuring Yarn-site.xml
• 5.8 Formatting Namenode
• 5.9 Starting HDFs
• 5.10 Check the startup situation

1. System Environment Description

CentOS 7.0 x64 version

192.168.1.7 Master
192.168.1.8 slave
192.168.1.9 slave
192.168.1.10 slave

2. Pre-installation preparations

2.1 CloseFirewall

# SYSTEMCTL Status Firewalld.service--View firewall status # systemctl stop Firewalld.service--Turn off firewall # systemctl disable FIREWALLD. Service--Permanently shut down the firewall

2.2 Check SSH installation, if not, install SSH

# SYSTEMCTL Status Sshd.service--View SSH status # Yum install Openssh-server openssh-clients

2.3 Installing VIM

# yum-y Install Vim

2.4 Setting a static IP address

# vim/etc/sysconfig/network-scripts/ifcfg-eno16777736

bootproto= "Static"
onboot= "Yes"
ipaddr0= "192.168.1.7"
prefix0= "255.255.255.0"
gateway0= "192.168.1.1"
dns1= "61.147.37.1"
Dns2= "101.226.4.6"

2.5 Modifying the host name

# vim/etc/sysconfig/network

Hostname=master

# vim/etc/hosts

192.168.1.7 master192.168.1.8 slave1192.168.1.9 slave2192.168.1.10 slave3

# Hostnamectl Set-hostname Master (CentOS7 the original modified host method is invalid)

2.6 Creating a Hadoop user

# useradd Hadoop--Create user named Hadoop # passwd Hadoop--Set a password for user Hadoop

2.7 Configuring SSH without key login

——— – Here are the actions above the master

 # su hadoop --Switch to Hadoop user $ cd ~      --Open User folder $  ssh-keygen -t rsa -P  '  --generate password pairs,/home/hadoop/.ssh/id_rsa and/home/hadoop/.ssh/id_ rsa.pub$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys         --add id_rsa.pub to the authorized key $ chmod 600 ~/.ssh/authorized_keys         --Modify Permissions $ su  --switch to root user # vim /etc/ssh/sshd_config    --modifying SSH profiles  RSAAuthentication yes  #启用RSA认证  pubkeyauthentication yes   #启用公钥私钥配对认证方式  AuthorizedKeysFile .ssh/authorized_keys  #公钥文件路径 # su hadoop - -Switch to Hadoop user $ scp ~/.ssh/id_rsa.pub [email protected]:~/                 --the public key to replicate on all slave machines 

———-below is the operation above the SLAVE1

# su Hadoop--Switch to Hadoop user $ mkdir ~/.ssh $ chmod ~/.ssh $ cat ~/id_rsa.pub >> ~/.ssh/authorized_keys --Append to authorization File "Authorized_keys" $ chmod ~/.ssh/authorized_keys--Modify permissions $ SU--switch back to root user # Vim/ Etc/ssh/sshd_config--Modify SSH config file rsaauthentication Yes #启用RSA认证 pubkeyauthentication Yes #启用公钥私钥配对认证方式 AUTHORIZEDKEYSF Ile. Ssh/authorized_keys #公钥文件路径

3. Install the necessary software

3.1 Installing the JDK

# RPM-IVH jdk-7u67-linux-x64.rpm

Preparing ...

##################################### [100%]

1:jdk

##################################### [100%]

Unpacking JAR files ...
Rt.jar ...
Jsse.jar ...
Charsets.jar ...
Tools.jar ...
Localedata.jar ...

# Vim/etc/profile Export java_home=/usr/java/jdk1.7.0_67 export path= $PATH: $JAVA _home/bin# Source profile-modification takes effect

3.2 Installing other required software

# yum install maven svn ncurses-devel gcc* lzo-devel zlib-devel autoconf automake libtool cmake openssl-devel

3.3 Installing Ant

# tar zxvf apache-ant-1.9.4-bin.tar.gz# vim/etc/profile export ant_home=/usr/local/apache-ant-1.9.4 export PATH= $PATH: $ANT _home/bin

3.4 Installing FindBugs

# tar zxvf findbugs-3.0.0.tar.gz# vim/etc/profile export findbugs_home=/usr/local/findbugs-3.0.0 export PATH= $PATH: $ Findbugs_home/bin

3.5 Installing PROTOBUF

# tar ZXVF protobuf-2.5.0.tar.gz (must be 2.5.0 version, otherwise error when compiling Hadoop) # CD protobuf-2.5.0#./configure--prefix=/usr/local# Make && make install

4. Compiling Hadoop source code

# tar zxvf hadoop-2.5.0-src.tar.gz# cd hadoop-2.5.0-src# mvn Package-pdist,native,docs-dskiptests-dtar

4.1 Maven central repository configuration (changed to Oschina, increased access speed)

# vim /usr/share/mavem/conf/settings.xml<mirrors>    <mirror>         <id>nexus-osc</id>         <mirrorOf>*</mirrorOf>        <name> Nexus osc</name>        <url>http://maven.oschina.net /content/groups/public/</url>    </mirror></mirrors><profiles>     <profile>    <id>jdk17</id>     <activation>        <activebydefault>true</ activebydefault>        <jdk>1.7</jdk>     </activation>    <properties>         <maven.compiler.source>1.7</maven.compiler.source>        < maven.compiler.target>1.7</maven.compiler.target>        < maven.compiler.compilerversion>1.7</maven.compiler.compilerversion>    </ Properties>            <repositories>            <repository>                 <id>nexus</id>                 <name>local  private nexus</name>                 <url>http://maven.oschina.net/content/groups/public/</url>                 <releases>                     <enabled>true</enabled>                 </releases>                 <snapshots>                      <enabled>false</enabled>                 </snapshots>             </repository>         </repositories >        <pluginRepositories>             <pluginrepository>                 <id>nexus</id>                 <name>local private nexus</name>                 <url>http://maven.oschina.net/ content/groups/public/</url>                 <releases>                     <enabled>true</enabled>                 </releases>                 <snapshots>                     <enabled>false</ enabled>                 </snapshots>            </ pluginrepository>         </pluginrepositories>     </profile></profiles>

4.2 After the compilation is complete, the directory/usr/hadoop-2.5.0-src/hadoop-dist/target/hadoop-2.5.0

# ./bin/hadoop versionhadoop 2.5.0subversion unknown -r unknowncompiled by  root on 2014-09-12t00:47zcompiled with protoc 2.5.0from source with  checksum 423dcd5a752eddd8e45ead6fd5ff9a24this command was run using /usr/ hadoop-2.5.0-src/hadoop-dist/target/hadoop-2.5.0/share/hadoop/common/hadoop-common-2.5.0.jar# file  lib//native/*lib//native/libhadoop.a:        current ar  Archivelib//native/libhadooppipes.a:   current ar archivelib//native/libhadoop.so:        symbolic link to  ' libhadoop.so.1.0.0 ' lib//native/ Libhadoop.so.1.0.0: elf 64-bit lsb shared object, x86-64, version 1   (SYSV),  dynamically linked, buildid[sha1]=0x972b31264a1ce87a12cfbcc331c8355e32d0e774,  not strippedlib//native/libhadooputils.a:   current ar archivelib//native/libhdfs.a:           current ar archivelib//native/libhdfs.so:          symbolic link to  ' libhdfs.so.0.0.0 ' lib//native/ Libhdfs.so.0.0.0:   elf 64-bit lsb shared object, x86-64, version  1  (SYSV),  dynamically linked, buildid[sha1]= 0x200ccf97f44d838239db3347ad5ade435b472cfa, not stripped

5. Configure Hadoop

5.1 Basic operations

# cp-r/usr/hadoop-2.5.0-src/hadoop-dist/target/hadoop-2.5.0/opt/hadoop-2.5.0# chown-r hadoop:hadoop/opt/ hadoop-2.5.0# vi/etc/profile Export hadoop_home=/opt/hadoop-2.5.0 export path= $PATH: $HADOOP _home/bin# su hadoop$ CD/ opt/hadoop-2.5.0$ mkdir-p dfs/name$ mkdir-p dfs/data$ mkdir-p tmp$ cd Etc/hadoop

5.2 Configuring all Slave nodes

$ vim Slavesslave1slave2slave3

5.3 Modifying hadoop-env.sh and yarn-env.sh

$ vim Hadoop-env.sh/vim yarn-env.shexport java_home=/usr/java/jdk1.7.0_67

5.4 Modifying Core-site.xml

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131702</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/opt/hadoop-2.5.0/tmp</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.hosts</name>
<value></value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.groups</name>
<value></value>
</property>
</configuration>

5.5 Modifying Hdfs-site.xml

<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/opt/hadoop-2.5.0/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/opt/hadoop-2.5.0/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:9001</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
</configuration>

5.6 Modifying Mapred-site.xml

# CP Mapred-site.xml.template Mapred-site.xml

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration>

5.7 Configuring Yarn-site.xml

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>768</value>
</property>
</configuration>

5.8 Formatting Namenode

$./bin/hdfs Namenode-format

5.9 Starting HDFs

$./sbin/start-dfs.sh$./sbin/start-yarn.sh

5.10 Check the startup situation

http://192.168.1.7:8088http://192.168.1.7:50070

Reference http://www.it165.net/admin/html/201403/2453.html

HADOOP-–2.2.0 4 64-bit source code compilation and distributed installation. PDF

Centos7-64bit compiled Hadoop-2.5.0, and distributed installation