Spark standalone Mode

It is very easy to install a spark cluster (StandaloneMode). In my example, I usedThreeMachines.

All machines run a OSUbuntu 12.04 32bit. One machine is named"Master", The other two are

Named"Node01"And"Node02"Respectively. The name of a machine can be set in:/Etc/hostname.

Further more, on every node (MACHINE), I useSameUser name.

 

1. On every node: install Java and setjava environment in ~ /. Bashrc:

# Set Java environment

Exportjava_home =/usr/local/jdk1.7.0 _ 67

Export jre_home = $ java_home/JRE

Export Path = $ java_home/bin: $ path

Exportclasspath =.: $ java_home/lib: $ jre_home/lib

Note that in my example, I usedjavaJdk1.7.0 _ 67And put it under /Usr/local.

2. On every node: Install spark.

Download any version of spark fromhttp: // spark.apache.org/downloads.html, in my example, I

ChoseSpark-1.1.0-bin-hadoop2.4.tgzAnd extract it/Usr/local.

3. Set upSSHSuch that every two nodesin the cluster can SSH each other without password. This step

Is also needed when you set up ahadoop cluster, there are abundant tutorials on the Internet, so

The details is omitted here.

4. On every node:

$ Sudo Vim/etc/hosts

And set the IP address of the nodesin the network. For example, I set the hosts file on every node:

127.0.0.1 localhost

223.3.86.xxx master

223.3.81.xxx node01

223.3.70.xxx node02

5. OnMasterNode: Enter the rootfolder of spark, and editCon/slaves. In my example:

$ CD/usr/local/spark-1.1.0-bin-hadoop2.4

$ Sudo Vim CONF/slaves

Edit slaves file:

Master

Node01

Node02

6. OnMasterNode: Enter the rootfolder of spark and start spark cluster.

$ CD/usr/local/spark-1.1.0-bin-hadoop2.4

$ Sbin/start-all.sh

7. Open http: // master: 8080/using your web browser to monitoring the cluster.

Spark standalone Mode