阿裡雲一鍵部署 Hadoop 分散式集群

摘要： Hadoop是一個能夠讓用戶輕鬆架構和使用的開源分散式運算架構，以一種可靠、高效、可伸縮的方式進行資料處理。通過ROS資源編排服務可以在阿裡雲上一鍵部署Hadoop集群。

Hadoop是一個能夠讓用戶輕鬆架構和使用的開源分散式運算架構，一種可靠、高效、可伸縮的方式進行資料處理。本文主要目的是為大家提供一種非常簡單的方法，在阿裡雲上部署Hadoop集群。
通過<阿裡雲ROS資源編排服務>，將VPC、NATGateway、ECS建立和Hadoop部署程序自動化，使大家能夠非常方便地部署一個Hadoop集群。本文建立的Hadoop集群包含三個節點：master.hadoop，slave1.hadoop，slave2.hadoop。

正常安裝Hadoop分為四步：設定ssh無密碼登入，安裝Java JDK，安裝設定Hadoop tar包，和啟動測試集群。

1. 設定ssh無密碼登入

集群中的所有主機應該可以無密碼ssh登入，因此3台ECS都應該執行如下指令，許諾所有的主機中臨時金鑰和公開金鑰均相同，這樣3台ECS就可以無密碼登入了。

"ssh-keygen -t rsa -P '' -f '/root/.ssh/id_rsa' 
",

"cd /root/.ssh 
",

"mv id_rsa bak.id_rsa 
",

"mv id_rsa.pub bak.id_rsa.pub 
",

"echo '-----BEGIN RSA PRIVATE KEY-----' > id_rsa 
",

"echo 'MIIEowIBAAKCAQEAzfQ/QHwWB1njU9+Wu3RYi9g+g5rydSpAE0klefTJuZjtcaic' >> id_rsa 
",

"echo 'SCeBN5avih8UToZ148+Ef2YzOtoosqluZpoYLCPSpAqr8pmviBJIU3vfu9mnDG9L' >> id_rsa 
",

"echo 'oevT6K8w3wCBRCmqu+vc0Tpju/EeCuYK3+8w7e2I6F3+zIYzhhX3qmkocje7ACnV' >> id_rsa 
",

"echo 'yh2DB/2m3sogTMc0RT+5y3kJAnC6TOIlGsYjOOkPbEF3Ifn1o4ZjOFOmQlcRJer4' >> id_rsa 
",

"echo '1E6rTdXAxS2uDNMFBCf9Xyx7O9J+ELTCAXc4h7AE6WLdQb5Apzv4t1KswCAtRenP' >> id_rsa 
",

"echo '1xGcUYY8I/JUT2VvBWtQJennrk9jrPZUFDcmcwIDAQABAoIBAAwzDZQaRYvF7UtI' >> id_rsa 
",

"echo 'kTslVyFhe8J76SS7jfQWfxvMPi66OkZjQG6duG+8g0VhNei42j7WSfjp6trvlT2P' >> id_rsa 
",

"echo '/7QgKJJkxNNmtmy2Ycljm9kmG0ibSePYq9g5ieHcjr6G3yFUfoKHJBtYpBO74pWu' >> id_rsa 
",

"echo 'rrI5DuLpERUCjFc9E8w7fOIhPH4XZ6wk/EmPxHTgxZk+aMvqptyPSbUyiUOvCiZf' >> id_rsa 
",

"echo 'MD0ircs9vgtslMVDlz9m6CoiNz6B3Yf6eLRoGGMiGnsQzZHIfnHCMX8i65Jc+TvQ' >> id_rsa 
",

"echo 'fLopIHzwBwI55xpcOIBRgYiEAQJhLsSNSFugoxMcwe9RalTGGS21HOQu4b3ZylKM' >> id_rsa 
",

"echo '8ofEVKECgYEA/Y4MzN04wAsM1yNuHN+9sdiVLG28LWH3dgpcXqa9gyNsWs9Gf8uf' >> id_rsa 
",

"echo 'qbuvQGeKDRXCW93wO1jO+pCYVrkyY3l+KhBKIqmkJT5gFsa8dBUvEBLALiHNg3+o' >> id_rsa 
",

"echo 'jR2Vqsemk8kMZA8zfJ3FKcMb1pt4S2GqepqsdC3DgzIIsxufCh7jSzsCgYEAz/Cv' >> id_rsa 
",

"echo 'Z7gAdSFC8q6QxFxSyhfZwGA6QW6ZU7rZv9ZdGySvZg+vHbpNVmQ0BAQzJui3Qs9u' >> id_rsa 
",

"echo 'XQMpYafXWzKsPzG6ZWvYXTF7fuxlovvG8Qd2A2QnGtGMB9YtQMHVqbsUDwxMjiTn' >> id_rsa 
",

"echo 'VBZILkDf+WCwQ98P5UMoumI0goIcNZ+AXhcqrikCgYBkSwLvKfYfqH9Mvfv5Odsr' >> id_rsa 
",

"echo '9NKUv1c20FB1BYYh/mxp6eIbTW/CbwXZup6IqCvoHxpBAlna77b3T6iibSDsTgtE' >> id_rsa 
",

"echo 'kirw6Q8/mBukBrpWZGa4QeJ4nPBQuncuUmx4H/7Y6CaZkZW5DiMF8OIbEmYT0y7+' >> id_rsa 
",

"echo 'zh222r9CLtFYH23aL/uSLwKBgQCS1xyG2eE41aw5RBznDWtJW15iA5If8sJD5ocu' >> id_rsa 
",

"echo 'eWp2aImUQS8ghxdmEozI6U5WA7CmdWUyObFXTPc/Z6FLXwqJ5IZ+CRt0neuIFNSA' >> id_rsa 
",

"echo 'EQy9iFQ1FBUW06BRQpBns7yOg9jr6BOTxchjIV0I9caDp1nKRIrWU9NQ9iCFnYVA' >> id_rsa 
",

"echo '7IsvQQKBgFxgF7UOhwaiMb/ATSuhm2v9kVvRPEO9umdo7YJ9I4L09lYbAtpcnusQ' >> id_rsa 
",

"echo 'fIROYL25VeEMgYcQyInc3Fm/sgJdbXQnUy+3QbbCcBCmcLj27LPQyxuu7p9hbPPT' >> id_rsa 
",

"echo 'Mxx37OmYvOkSVTQz0T9HfDGvOJgt4t4cXD4T/7ewk62p6jdpSQrt' >> id_rsa 
",

"echo '-----END RSA PRIVATE KEY-----' >> id_rsa 
",

"echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDN9D9AfBYHWeNT35a7dFiL2D6DmvJ1KkATSSV59Mm5mO1xqJxIJ4E3lq+KHxROhnXjz4R/ZjM62iiyqW5mmhgsI9KkCqvyma+IEkhTe9+72acMb0uh69PorzDfAIFEKaq769zROmO78R4K5grf7zDt7YjoXf7MhjOGFfeqaShyN7sAKdXKHYMH/abeyiBMxzRFP7nLeQkCcLpM4iUaxiM46Q9sQXch+fWjhmM4U6ZCVxEl6vjUTqtN1cDFLa4M0wUEJ/1fLHs70n4QtMIBdziHsATpYt1BvkCnO/i3UqzAIC1F6c/XEZxRhjwj8lRPZW8Fa1Al6eeuT2Os9lQUNyZz root@iZ2zee53wf4ndvajz30cdvZ' > id_rsa.pub 
",

"cp id_rsa.pub authorized_keys 
",

"chmod 600 authorized_keys 
",

"chmod 600 id_rsa 
",

"sed -i 's/#StrictHostKeyChecking ask/StrictHostKeyChecking no/' /etc/ssh/ssh_config 
",

"sed -i 's/GSSAPIAuthentication yes/GSSAPIAuthentication no/' /etc/ssh/ssh_config 
",

"service sshd restart 
",

為了許諾安全，防止對外洩露金鑰和公開金鑰。我們要在master上執行下述指令，取代掉公用的臨時金鑰與公開金鑰：

"scp -r /root/.ssh/bak.*root@$ipaddr_slave1:/root/.ssh/
",

"scp -r /root/.ssh/bak.*root@$ipaddr_slave2:/root/.ssh/
",

"ssh root@$ipaddr_slave1 "cd /root/.ssh; rm -f id_rsa; mv bak.id_rsa id_rsa; rm -f id_rsa.pub; mv bak.id_rsa.pub id_rsa.pub; rm -f authorized_keys; cp id_rsa.pub authorized_keys; chmod 600 authorized_keys; chmod 600 id_rsa; exit" 
",

"ssh root@$ipaddr_slave2 "cd /root/.ssh; rm id_rsa; mv bak.id_rsa id_rsa; rm id_rsa.pub; mv bak.id_rsa.pub id_rsa.pub; rm authorized_keys; cp id_rsa.pub authorized_keys; chmod 600 authorized_keys; chmod 600 id_rsa; exit" 
",

"cd /root/.ssh 
 rm id_rsa 
 mv bak.id_rsa id_rsa 
 rm id_rsa.pub 
 mv bak.id_rsa.pub id_rsa.pub 
 rm authorized_keys 
 cp id_rsa.pub authorized_keys 
 chmod 600 authorized_keys 
 chmod 600 id_rsa 
 cd 
",

2. 安裝設定JDK

master上安裝JDK，並遠端控制在slaves上安裝JDK。

"yum -y install aria2 
",

"aria2c --header='Cookie: oraclelicense=accept-securebackup-cookie' $JdkUrl 
",

"mkdir -p $JAVA_HOME 
tar zxvf jdk*-x64.tar.gz -C $JAVA_HOME 
cd $JAVA_HOME 
mv jdk*.** ./ 
rmdir hadoop-*.* 
",

"echo>> /etc/profile 
",

"echo export HADOOP_HOME=$HADOOP_HOME >> /etc/profile 
",

"echo export HADOOP_MAPRED_HOME=$HADOOP_HOME >> /etc/profile 
",

"echo export HADOOP_COMMON_HOME=$HADOOP_HOME >> /etc/profile 
",

"echo export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native >> /etc/profile 
",

"echo export YARN_HOME=$HADOOP_HOME >> /etc/profile 
",

"echo export HADOOP_HDFS_HOME=$HADOOP_HOME >> /etc/profile 
",

"echo export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop >> /etc/profile 
",

"echo export HADOOP_OPTS=-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_HOME/lib/native >> /etc/profile 
",

"echo export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH >> /etc/profile 
",

"echo export CLASSPATH=.:$CLASSPATH:$HADOOP__HOME/lib >> /etc/profile 
",

"source /etc/profile 
",

" 
",

"mkdir -p $HADOOP_HOME/tmp 
",

"mkdir -p $HADOOP_HOME/dfs/name 
",

"mkdir -p $HADOOP_HOME/dfs/data 
",

###

此處要對這4個檔案進行正確設定，具體內容請參考範本Hadoop_Distributed_Env_3_ecs.json.

core-site.xml，hdfs-site.xml，mapred-site.xml，yarn-site.xml

###

"ssh root@$ipaddr_slave1 "mkdir -p $HADOOP_HOME; exit" 
",

"ssh root@$ipaddr_slave2 "mkdir -p $HADOOP_HOME; exit" 
",

"scp -r $HADOOP_HOME/*root@$ipaddr_slave1:$HADOOP_HOME 
",

"scp -r $HADOOP_HOME/*root@$ipaddr_slave2:$HADOOP_HOME 
",

" 
",

"cd $HADOOP_HOME/etc/hadoop 
",

"echo $ipaddr_slave1 > slaves 
",

"echo $ipaddr_slave2 >> slaves 
",

4. 啟動測試集群

最後格式化HDFS，關閉防火牆，啟動集群。

"hadoop namenode -format 
",

"systemctl stopfirewalld 
",

"start-dfs.sh 
",

"start-yarn.sh 
",

一鍵部署Hadoop集群>>

注意：

·必須確保可以正確下載Jdk和Hadooptar 包，我們可以選擇類似如下的URL：

·http://download.oracle.com/otn-pub/java/jdk/8u121-b13/e9e7ea248e2c4826b92b3f075a80e441/jdk-8u121-linux-x64.tar.gz(Must set 'Cookie:oraclelicense=accept-securebackup-cookie' in Header, when download.)

·http://mirrors.hust.edu.cn/apache/hadoop/core/hadoop-2.7.1/hadoop-2.7.1.tar.gz

·利用該範本建立時，只能選擇CentOS系統；

·為了防止Timeout 失敗，可以設定為120分鐘；

·我們選擇的資料中心在上海。

建立完成後，查看資源棧概況：

瀏覽器中輸入圖中的的WebsiteUrl，得到如下結果，則部署成功：

Hadoop_Distributed_Env_3_ecs.json：通過該範本可以一鍵部署上面的集群。
Hadoop_Distributed_ecsgroup.json: 該範本允許用戶指定slaves節點的數量。

相關產品：

1. 資源編排ROS