Spark on Yarn deployment

Environment: Linux, 8 GB memory, 60 GB hard disk, hadoop 2.2.0, spark 1.0.0, scala 2.10.3

1. Install the hadoop yarn Cluster

Http://blog.csdn.net/zlcd1988/article/details/36008681

This blog describes in detail how to deploy a hadoop yarn cluster.

2. Install Scala

Http://www.scala-sbt.org/download.html

Download scala-2.10.3.tgz

$ tar -zxvf scala-2.10.3.tgz$ mv scala-2.10.3 /usr/scala

3. Install spark

$ wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.0-bin-hadoop2.tgz$ tar -zvxf spark-1.0.0-bin-hadoop2.tgz$ mv spark-1.0.0-bin-hadoop2 /usr/spark

4. Configure Environment Variables

In ~ /. Bash_profile add Environment Variables

Export spark_home =/usr/spark
Export scala_home =/usr/Scala
Export yarn_home = $ hadoop_home
Export Path = $ path: $ scala_home/bin: $ spark_home/bin
Export spark_jar = $ spark_home/lib/spark-assembly-1.0.0-hadoop2.2.0.jar
Export classpath =.: $ java_home/lib/dt. jar: $ java_home/lib/tools. Jar

Run source. bash_profile to make the environment variable take effect immediately.

5. Edit $ spark_home/CONF/Add at the end of the spark-env.sh:

Export spark_home =/usr/spark
Export spark_jar =/usr/spark/lib/spark-assembly-1.0.0-hadoop2.2.0.jar

6. Test whether spark can run on yarn.

$ spark-class org.apache.spark.deploy.yarn.Client --jar /usr/spark/lib/spark-examples-1.0.0-hadoop2.2.0.jar --class org.apache.spark.examples.JavaSparkPi --args yarn-standalone --num-workers 1 --master-memory 1G --worker-memory 1G --worker-cores 1