Running spark based on Mesos

Background introduction

Spark has multiple cluster operating modes, for example: Standalone,yarn,mesos.
Here's how to run spark on Mesos, which is also the official recommended way to run.
Let's start with a brief introduction to Mesos before running spark.
Mesos Computing Framework is a cluster manager that provides efficient, cross-distributed application or framework resource isolation and sharing, and can run Haoop,spark,marathon and many other frameworks. Using zookeeper to implement fault tolerance mechanism, use Linux containers to isolate tasks, support a variety of resource computing allocations, based on mesos we can achieve a variety of services mix, finer granularity, more efficient utilization of resources, to avoid the waste of resources.
Installation details about Mesos do not repeat here, details can be viewed Http://get.dataman.io.
is the working mode of the spark cluster, when we use Mesos Mesos Master will replace the spark master as the cluster manager.

environment with Ubuntu 14.04 as an example

1. Preparatory work

(1) Installing JDK 1.7
(2) Installing Scala 2.11
(3) Install Libcurl4-nss-dev (Note: This is a mistake I encountered during my own installation)
(4) Download spark binary package http://spark.apache.org/downloads.html
(5) Spark driver can be installed on any machine, if the installation of Spark driver is already installed Mesos, you can find/usr/local/lib/libmesos.so this is the driver that spark mesos need to load, If Mesos is not installed on the machine, it can be copied from the machine where the Mesos is installed.

2. Configure Spark Driver

(1) Download spark binary package

wget http://mirror.reverse.net/pub/apache/spark/spark-1.4.0/spark-1.4.0-bin-hadoop2.6.tgz

(2) Decompression

tar -zxf spark-1.4.0-bin-hadoop2.6.tgz

(3) Enter the catalogue

cd spark-1.4.0-bin-hadoop

(4) Copy the configuration file

cp conf/spark-defaults.conf.template conf/spark-defaults.conf && cp conf/spark-env.sh.template conf/spark-env.sh

(5) Modify the spark-env.sh configuration options, I mainly modified the following configuration

  export Mesos_native_java_library=/usr/local/lib/libmesos.so # Loading the path to the Mesos library export Master=mesos://zk://192.168.100.7:2181,192.168.100.8:2181,192.168.100.9:2181/mesos # If a single master node can be used Mesos://ip:host, if it is a zookeeper managed multi-master node will need to mesos://zk:// 192.168.100.7:2181,192.168.100.8:2181,192.168.100.9:2181/mesos export spark_local_ip= ' ifconfig eth0 | awk '/inet addr/{print substr ($2,6)} ' export spark_local_hostname= ' ifconfig eth0 | awk '/inet addr/{print substr ($2,6)} ' export java_home=$ (readlink-f/usr/bin/java | sed "S:jre/bin/java::") export SPARK _executor_uri=<url of spark-1.4.0.tar.gz uploaded above> #这个是spark二进制包所在的位置, when Mesos runs the task at the compute node, The compute node must have a binary package of spark to establish the spark's operating environment. 

(6) Run Spark-shell Now the configuration of the spark driver is basically configured, we can run the Spark-shell in the Bin directory and view the Mesos Web page active after the successful run Frameworks if it comes out as shown in the process, it means that we are already configured successfully.
Now we can write our own spark program.

Running spark based on Mesos