Hadoop介紹及最新穩定版Hadoop 2.4.1下載地址及單節點安裝

標籤：des   style   http   color   os   io   java   strong   ar   

 Hadoop介紹

    Hadoop是一個能對大量資料進行分散式處理的軟體架構。其基本的組成包括hdfsDistributed File System和可以運行在hdfs檔案系統上的MapReduce編程模型，以及基於hdfs和MapReduce而開發的一系列上層應用軟體。

    hdfs是在一個網路中以流式資料訪問模式來儲存超大檔案的跨越多台電腦的Distributed File System。目前支援的超大檔案的範圍為從MB級至PB級。

    MapReduce是一種可用於資料處理的編程模型，基於MapReduce模型的程式本質上都是並行啟動並執行。基於MapReduce編程模型的程式包括完成資料提取的map函數，對中間結果進行處理的merge函數（merge函數一般是可選的），以及產生最終處理結果的reduce函數。經過map函數和merge函數進行處理後的資料將是經過排序和分組的key-value，經過reduce對這些中間結果處理後產生最終的計算結果。其中map函數都是並行啟動並執行，每個map函數負責處理大檔案的一個檔案塊，因此對於基於hdfs檔案系統的大檔案來說，map函數可以充分利用多台電腦的處理能力，快速計算並出中間結果。

 

     The Apache™ Hadoop® project develops open-source software for reliable, scalable, distributed computing.

     The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

    The project includes these modules:

Hadoop Common: The common utilities that support the other Hadoop modules.

Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data.

Hadoop YARN: A framework for job scheduling and cluster resource management.

Hadoop MapReduce: A YARN-based system for parallel processing of large data sets. www.169it.com

 

Hadoop最新穩定版Hadoop 2.4.1

Hadoop 2.4.1(2.X系列穩定版):  Hadoop 2.4.1下載   

 

 

hadoop 2.4.1 虛擬機器安裝-單節點安裝步驟

1 安裝java及java 環境變數的設定  

2 設定賬戶，主機的hostname /etc/hosts

 使用者的.bash_profile 中加入如下內容

12345

export JAVA_HOME=/usr/java/jdk1.7.0_60export HADOOP_PREFIX=/home/hadoop/hadoop-2.4.1export CLASSPATH=".:$JAVA_HOME/lib:$CLASSPATH"export PATH="$JAVA_HOME/:$HADOOP_PREFIX/bin:$PATH"export HADOOP_PREFIX PATH CLASSPATH

3 設定 無密碼登陸

先確保所有主機的防火牆處於關閉狀態。 

1234	$cd ~/.ssh$ssh-keygen -t rsa    #然後一直按斷行符號鍵，就會按照預設的選項將產生的密鑰儲存在.ssh/id_rsa檔案中。$cp id_rsa.pub authorized_keyssudo service sshd restart

4 hadoop2.4.1 的配置 

進入hadoop2.4.1檔案夾，配置etc/hadoop中的檔案。

12	hadoop-env.shexport JAVA_HOME=/usr/java/jdk1.7.0_60

另外可選的添加上：

12	export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/nativeexport HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib"

 

5 配置hadoop2.4.1的core-site.xml 

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455

<configuration>       <property>                <name>fs.default.name</name>                <value>hdfs://localhost:9000</value>       </property>       <property>                <name>io.file.buffer.size</name>                <value>131072</value>        </property>       <property>               <name>hadoop.tmp.dir</name>               <value>file:/home/hadoop/tmp</value>       </property>        <property>                <name>dfs.namenode.name.dir</name>               <value>file:/home/hadoop/hadoop-2.4.1/dfs/name</value>       </property>       <property>               <name>dfs.datanode.data.dir</name>               <value>file:/home/hadoop/hadoop-2.4.1/dfs/data</value>       </property></configuration>hdfs-site.xml <configuration>        <property>                <name>dfs.namenode.name.dir</name>               <value>file:/home/hadoop/hadoop-2.4.1/dfs/name</value>       </property>       <property>               <name>dfs.datanode.data.dir</name>               <value>file:/home/hadoop/hadoop-2.4.1/dfs/data</value>       </property>       <property>               <name>dfs.replication</name>               <value>1</value>       </property></configuration>mapred-site.xml<configuration>       <property>               <name>mapreduce.jobtracker.address</name>               <value>hdfs://localhost:9001</value>       </property></configuration>yarn-site.xml<configuration>       <property>               <name>mapreduce.framework.name</name>               <value>yarn</value>       </property>       <property>               <name>yarn.nodemanager.aux-services</name>               <value>mapreduce_shuffle</value>       </property></configuration>

 

經過以上五步，hadoop2.4.1單機環境配置都已經完成了，下面啟動:

./bin/hadoop namenode –format    格式化結點資訊

bin/start-all.sh.     新版本的hadoop其實不建議這麼直接start-all，建議一步步來，先start-dfs，然後在start-map

./bin/hadoop dfsadmin -report

http://localhost:50070 

 

本文來源:Hadoop介紹及最新穩定版Hadoop 2.4.1及單節點安裝

Hadoop介紹及最新穩定版Hadoop 2.4.1及單節點安裝