RedHat Ent 6.5 64bit編譯安裝hadoop2.4.1

標籤：style   blog   http   color   os   io   檔案   2014   

RedHat Ent 6.5 64bit編譯安裝hadoop2.4.1

感謝原帖：http://blog.csdn.net/w13770269691/article/details/16883663/

  step 1.修改yum：（針對redhat ent未註冊使用者，註冊使用者直接跳過這一步）

    參考：http://blog.csdn.net/zhngjan/article/details/20843465

  step 2.下載源碼包：http://mirrors.hust.edu.cn/apache/hadoop/common/stable2/

wget http://mirrors.hust.edu.cn/apache/hadoop/common/stable2/hadoop-2.4.1-src.tar.gz

  step 3.準備maven作為編譯hadoop的工具

    a.下載編譯好的maven包：

wget http://mirror.bit.edu.cn/apache/maven/maven-3/3.1.1/binaries/apache-maven-3.1.1-bin.tar.gz
tar -zxvf apache-maven-3.1.1-bin.tar.gz -C /opt/

    b.配置maven的環境變數，在/etc/profile檔案結尾中添加如下代碼

export MAVEN_HOME=/opt/apache-maven-3.1.1export PATH=$PATH:${MAVEN_HOME}/bin

    c.執行如下命令使設定檔生效

source /etc/profile

    d.測試maven

mvn -version

   e.由於maven國外伺服器可能連不上，先給maven配置一下國內鏡像

    在maven目錄下，conf/settings.xml,在<mirrors></mirros>裡添加如下內容（注意不要添加到注釋裡面了）

<mirror>   <id>nexus-osc</id>    <mirrorOf>*</mirrorOf>    <name>Nexusosc</name>    <url>http://maven.oschina.net/content/groups/public/</url>  </mirror>

 在maven目錄下，conf/settings.xml,在<profiles></profiles>添加如下內容（注意不要添加到注釋裡面了）

<profile>    <id>jdk-1.7</id>    <activation>    <jdk>1.7</jdk>    </activation>    <repositories>    <repository>     <id>nexus</id>     <name>local private nexus</name>     <url>http://maven.oschina.net/content/groups/public/</url>     <releases>      <enabled>true</enabled>     </releases>     <snapshots>      <enabled>false</enabled>     </snapshots>    </repository>    </repositories>    <pluginRepositories>     <pluginRepository>      <id>nexus</id>      <name>local private nexus</name>      <url>http://maven.oschina.net/content/groups/public/</url>      <releases>        <enabled>true</enabled>      </releases>      <snapshots>        <enabled>false</enabled>      </snapshots>     </pluginRepository>    </pluginRepositories>  </profile>  

  step 4.hadoop2.4.1編譯需要protoc2.5.0的支援，所以還要安裝下載protoc2.5.0

    官方網址：https://code.google.com/p/protobuf/downloads/list

    百度網盤網址：http://pan.baidu.com/s/1pJlZubT

    a.對protoc進行編譯安裝前先要裝幾個依賴包:gcc,gcc-c++,make 如果已經安裝的可以忽略

yum install gcc  yum intall gcc-c++  yum install make 

   b.配置protoc

tar -xvf protobuf-2.5.0.tar.bz2  cd protobuf-2.5.0  ./configure --prefix=/opt/protoc/  make && make install yum install gcc 

    紅色字型是自訂的protoc安裝目錄，完成上面的命令後，再配置protoc環境變數，同樣在/etc/profile尾部加入：

export PATH=/opt/protoc/bin:$PATH

  step 5.，還不要著急開始編譯安裝，不然又是各種錯誤,需要安裝cmake,openssl-devel,ncurses-devel依賴

yum install cmake  yum install openssl-devel  yum install ncurses-devel  

  step 6. 解壓並編譯hadoop源碼(一定要進入到源碼解壓的位置再執行mvn編譯)

tar -zxvf hadoop-2.4.1-src.tar.gz -C /opt/
cd /opt/hadoop-2.4.1-src
mvn package -Pdist,native -DskipTests -Dtar

編譯耗時比較長，耐心等待，成功後結果類似於：

 pache Hadoop Distributed Copy .................... SUCCESS [33.648s][INFO] Apache Hadoop Archives ............................ SUCCESS [7.303s][INFO] Apache Hadoop Rumen ............................... SUCCESS [21.288s][INFO] Apache Hadoop Gridmix ............................. SUCCESS [14.611s][INFO] Apache Hadoop Data Join ........................... SUCCESS [8.334s][INFO] Apache Hadoop Extras .............................. SUCCESS [10.737s][INFO] Apache Hadoop Pipes ............................... SUCCESS [18.321s][INFO] Apache Hadoop OpenStack support ................... SUCCESS [17.136s][INFO] Apache Hadoop Client .............................. SUCCESS [14.563s][INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.254s][INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [17.245s][INFO] Apache Hadoop Tools Dist .......................... SUCCESS [14.478s][INFO] Apache Hadoop Tools ............................... SUCCESS [0.084s][INFO] Apache Hadoop Distribution ........................ SUCCESS [41.979s][INFO] ------------------------------------------------------------------------[INFO] BUILD SUCCESS[INFO] ------------------------------------------------------------------------[INFO] Total time: 34:24.464s[INFO] Finished at: Thu Aug 07 14:25:51 CST 2014[INFO] Final Memory: 159M/814M[INFO] ------------------------------------------------------------------------You have new mail in /var/spool/mail/root

編譯後的路徑在:hadoop-2.4.1-src/hadoop-dist/target/hadoop-2.4.1

進入hadoop-2.4.1目錄，測試安裝是否成功

[[email protected] bin]# cd /opt/hadoop-2.4.1-src/hadoop-dist/target/hadoop-2.4.1/bin
[[email protected] bin]# ./hadoop version
Hadoop 2.4.1
Subversion Unknown -r Unknown
Compiled by root on 2014-08-07T05:52Z
Compiled with protoc 2.5.0
From source with checksum bb7ac0a3c73dc131f4844b873c74b630
This command was run using
/opt/hadoop-2.4.1-src/hadoop-dist/target/hadoop-2.4.1/share/hadoop/common/hadoop-common-2.4.1.jar
[[email protected] bin]# cd ..
[[email protected] hadoop-2.4.1]# file lib/native/*
lib/native/libhadoop.a: current ar archive
lib/native/libhadooppipes.a: current ar archive
lib/native/libhadoop.so: symbolic link to `libhadoop.so.1.0.0‘
lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1
(SYSV), dynamically linked, not stripped
lib/native/libhadooputils.a: current ar archive
lib/native/libhdfs.a: current ar archive
lib/native/libhdfs.so: symbolic link to `libhdfs.so.0.0.0‘
lib/native/libhdfs.so.0.0.0: ELF 64-bit LSB shared object, x86-64, version 1
(SYSV), dynamically linked, not stripped
[[email protected] hadoop-2.4.1]#