Installation of spark-1.1.0-bin-hadoop2.4.tgz

Apache spark1.1.0 deployment and development environment setup-Mark Lin

0. Prepare

For the purpose of learning, this article deploys spark in a virtual machine. Install the following software on a virtual machine:

• Ubuntu 14.04
• Hadoop-2.4.0.tar.gz
• Jdk-7u67-linux-x64.tar.gz
• Scala-2.10.4.tgz
• Spark-1.1.0-bin-hadoop2.4.tgz

1. Install JDK

....

2. install and configure SSH

....

3. Install hadoop2.4.0

....

4. Install Scala

@ Download the compiled spark file and view the Pom. xml file to get the scala version;

@ Download scala:

@ Decompress and set the environment variables:

Source/etc/profile

@ Test scala:

[[email protected] local]# source /etc/profile[[email protected] local]# scala -versionScala code runner version 2.10.4 -- Copyright 2002-2013, LAMP/EPFL

5. Install spark

@ Decompress and set Environment Variables

@ $ Spark_home/CONF/spark-env.sh

export SCALA_HOME=/usr/local/scalexport JAVA_HOME=/usr/local/jdkexport SPARK_MASTER_IP=localhostexport SPARK_WORKER_MEMORY=1000m

6. Spark startup and testing:

Start:

# To the spark installation directory: sbin/start-all.sh

Test:

bin/run-example SparkPi

Installation of spark-1.1.0-bin-hadoop2.4.tgz