spark範例測試

spark on yarn計算平台已經搭建完成,下面來利用這個平台運行下官方的範例,範例全部在/home/work/spark目錄下執行 SparkPi

./bin/spark-submit –class org.apache.spark.examples.SparkPi –master yarn –deploy-mode cluster examples/jars/spark-examples*.jar 10

SparkPageRank

首先編寫一個檔案,內容如下:

1 21 31 42 13 11 52 5

將其添加到hdfs檔案系統中hadoop fs -put -R test.txt /user/work/

./bin/spark-submit –class org.apache.spark.examples.SparkPageRank –master yarn –deploy-mode cluster examples/jars/spark-examples*.jar test.txt

計算結果如下

4 has rank: 0.250559973109555.5 has rank: 0.35684113180533983.2 has rank: 0.250559973109555.3 has rank: 0.250559973109555.1 has rank: 0.46884347608735455.

CustomReceiver

spark01中執行nc -lk 9999

spark00中執行./bin/spark-submit --class org.apache.spark.examples.streaming.CustomReceiver --master yarn --deploy-mode cluster examples/jars/spark-examples*.jar 10.19.145.213 9999