大資料：從入門到XX(四)

標籤：大資料 單機版 偽分布式版 redhat6.4 hadoop2.7.2 ssh rsync http 保護模式 0660 644

    Hadoop Local (Standalone) Mode（單機版）的安裝純粹只是為練手，練完了單機版，下面該練練Pseudo-Distributed Mode（偽分布式版）的安裝了。偽分布式是在一台物理機上類比實現了hadoop的全部功能。包括ssh訪問、hdfs格式化、mapReduce執行、yarn資源管理等，偽分布式安裝是單機版安裝的繼續，部分內容依賴於單機版的安裝情況。

    1、首先確認在redhat6.4上有沒有安裝SSH。

[[email protected] ~]# rpm -qa|grep ssh openssh-askpass-5.3p1-81.el6.x86_64 trilead-ssh2-213-6.2.el6.noarch openssh-clients-5.3p1-81.el6.x86_64 ksshaskpass-0.5.1-4.1.el6.x86_64 openssh-server-5.3p1-81.el6.x86_64 libssh2-1.2.2-7.el6_2.3.x86_64 openssh-5.3p1-81.el6.x86_64

    2、確認有沒有安裝 rsync

[[email protected] ~]# rpm -qa|grep rsync rsync-3.0.6-9.el6.x86_64

    3、執行命令，測試ssh是否能夠通過無密碼訪問

[[email protected] ~]$ ssh localhost The authenticity of host ‘localhost (::1)‘ can‘t be established. RSA key fingerprint is 05:9e:ac:46:24:aa:c1:45:be:f6:55:83:10:6d:45:6d. Are you sure you want to continue connecting (yes/no)?

    說明：如果每次都需要輸入密碼，則說明沒有配置公開金鑰、私密金鑰。

    4、配置ssh，產生公開金鑰、私密金鑰

[[email protected] ~]$ ssh-keygen -t dsa -P ‘‘ -f ~/.ssh/id_dsa Generating public/private dsa key pair. Your identification has been saved in /home/hadoop/.ssh/id_dsa. Your public key has been saved in /home/hadoop/.ssh/id_dsa.pub. The key fingerprint is: d4:fc:32:6f:5c:d6:5a:47:89:8a:9d:79:d1:b5:51:14 [email protected] The key‘s randomart image is: +--[ DSA 1024]----+ |               E*| |         o    o =| |        . o  o +.| |       .  o.+ .o | |        S.o=..o +| |           =.o o.| |            + .  | |           .     | |                 | +-----------------+ 執行下面的命令合并公開金鑰。 [[email protected] ~]$ cat ~/.ssh/id_dsa.pub >>~/.ssh/authorized_keys 執行下面的命令修改公開金鑰檔案模式。 [[email protected] .ssh]$ chmod 644 authorized_keys 這裡需要說明一下，官方文檔是基於ubuntu做的說明，要求執行chmod 0660 ~/.ssh/authorized_keys，但是在redhat6.4上肯定只能執行chmod 644 authorized_keys，否則會出錯。

    5、在設定檔中設定JAVA_HOME

[[email protected] ~]$ vi hadoop-2.7.2/etc/hadoop/hadoop-env.sh # set to the root of your Java installation export JAVA_HOME=/usr/java/jdk1.8.0_92

    6、配置core-site.xml

vi hadoop-2.7.2/etc/hadoop/core-site.xml <configuration>     <property>         <name>fs.defaultFS</name>         <value>hdfs://localhost:9000</value>     </property> </configuration>

    7、配置hdfs-site.xml

vi hadoop-2.7.2/etc/hadoop/hdfs-site.xml <configuration>     <property>         <name>dfs.replication</name>         <value>1</value>     </property> </configuration>

    8、格式化namenode

[[email protected] hadoop-2.7.2]$ bin/hdfs namenode -format 16/03/12 19:21:50 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG:   host = localhost/127.0.0.1 STARTUP_MSG:   args = [-format] STARTUP_MSG:   version = 2.7.2 。。。 。。。 16/03/12 19:21:55 INFO common.Storage: Storage directory /tmp/hadoop-hadoop/dfs/name has been successfully formatted. 16/03/12 19:21:56 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 16/03/12 19:21:56 INFO util.ExitUtil: Exiting with status 0 16/03/12 19:21:56 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1 ************************************************************/

    9、啟動hdfs

[[email protected] sbin]$ start-dfs.sh 16/03/12 20:04:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Starting namenodes on [localhost] localhost: starting namenode, logging to /home/hadoop/hadoop-2.7.2/logs/hadoop-hadoop-namenode-localhost.localdomain.out localhost: starting datanode, logging to /home/hadoop/hadoop-2.7.2/logs/hadoop-hadoop-datanode-localhost.localdomain.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /home/hadoop/hadoop-2.7.2/logs/hadoop-hadoop-secondarynamenode-localhost.localdomain.out 16/03/12 20:04:57 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

    10、確認是否能成功訪問hdfs的web頁面

http://localhost:50070/

650) this.width=650;" src="http://s2.51cto.com/wyfs02/M01/83/3D/wKiom1dtT6LTfuLaAACZ7Ishbc8521.png-wh_500x0-wm_3-wmp_4-s_254652364.png" title="http-50070-part.png" alt="wKiom1dtT6LTfuLaAACZ7Ishbc8521.png-wh_50" />

    11、將本地檔案匯入hdfs中，測試mapReduce示範程式

[[email protected] sbin]$ hdfs dfs -mkdir /user 16/03/12 20:46:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [[email protected] hadoop-2.7.2]$ hdfs dfs -put ./etc/hadoop/ /user [[email protected] hadoop-2.7.2]$ hadoop jar ~/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar grep /user/hadoop output ‘de[a-z.]+‘ 說明：下面這條語句是在hdfs上，直接查看輸出 [[email protected] sbin]$ hdfs dfs -cat /user/hadoop/output/* 說明：下面這條語句是將輸出內容從hdfs中拷貝到本地檔案夾下 [[email protected] output]$ hdfs dfs -get /user/hadoop/output output 說明：查看本地檔案夾下的內容 [[email protected] ~]$ cat output/* 160     description 128     der 63      der. 31      default 。。。 。。。

    說明一下，上面的語句建立“/user”目錄如果失敗，可能是因為目錄啟用了保護模式，需要先執行以下命令：[[email protected] sbin]$ hadoop dfsadmin -safemode leave

    12、停止hdfs

[[email protected] sbin]$ stop-dfs.sh 16/03/12 20:09:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Stopping namenodes on [localhost] localhost: stopping namenode localhost: stopping datanode Stopping secondary namenodes [0.0.0.0] 0.0.0.0: stopping secondarynamenode 16/03/12 20:09:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

    13、在單節點上啟用YARN，配置mapred-site.xml

在hadoop2.7.2發布版中，沒有找到mapred-site.xml檔案，所以直接從模板複製過來一份。 [[email protected] sbin]$ cp mapred-site.xml.template mapred-site.xml vi etc/hadoop/mapred-site.xml <configuration>     <property>         <name>mapreduce.framework.name</name>         <value>yarn</value>     </property> </configuration>

    14、配置yarn-site.xml

vi  etc/hadoop/yarn-site.xml <configuration>     <property>         <name>yarn.nodemanager.aux-services</name>         <value>mapreduce_shuffle</value>     </property> </configuration>

    15、啟動yarn

[[email protected] sbin]$ start-yarn.sh starting yarn daemons starting resourcemanager, logging to /home/hadoop/hadoop-2.7.2/logs/yarn-hadoop-resourcemanager-localhost.localdomain.out localhost: starting nodemanager, logging to /home/hadoop/hadoop-2.7.2/logs/yarn-hadoop-nodemanager-localhost.localdomain.out

    16、訪問yarn的web頁面

http://localhost:8088/

650) this.width=650;" src="http://s5.51cto.com/wyfs02/M00/83/3D/wKiom1dtT3_TiVnYAAEgyCs9T-M941.png-wh_500x0-wm_3-wmp_4-s_3383490219.png" title="http-8088-part.png" alt="wKiom1dtT3_TiVnYAAEgyCs9T-M941.png-wh_50" />

    17、停止yarn

[[email protected] sbin]$ stop-yarn.sh stopping yarn daemons stopping resourcemanager localhost: stopping nodemanager no proxyserver to stop

以上是偽分布式hadoop的安裝，整個過程基本遵照hadoop官方文檔執行，在執行過程中，如果碰到其它問題，大多都是由於作業系統引起的，比如系統軟體的安裝、網路設定等情況。

本文出自 “沈進群” 部落格，謝絕轉載！

大資料：從入門到XX(四)