Spark 1.6.1 Distributed cluster Environment setup

First, software preparation

Scala-2.11.8.tgz

Spark-1.6.1-bin-hadoop2.6.tgz

Second, Scala installation 1, Master machine

(1) Download scala-2.11.8.tgz, extract to/opt directory, namely:/opt/scala-2.11.8.

(2) Modify the user and user groups that the scala-2.11.8 directory belongs to.

?

1	sudo chown -R hadoop:hadoop scala-2.11.8

(3) Modify the environment variable file. BASHRC, add the following content.

?

123	# Scala Envexport SCALA_HOME=/opt/scala-2.11.8export PATH=$PATH:$SCALA_HOME/bin

Run the source. BASHRC to make the environment variable effective.

(4) Verifying the Scala installation

2. Slave machine

SLAVE01 and SLAVE02 are installed by referring to the master machine installation steps.

Third, Spark installation 1, Master machine

(1) Download spark-1.6.1-bin-hadoop2.6.tgz, unzip to the/OPT directory.

(2) Modify the user and user groups that the spark-1.6.1-bin-hadoop2.6 directory belongs to.

?

1	sudo chown -R hadoop:hadoop spark-1.6.1-bin-hadoop2.6

(3) Modify the environment variable file. BASHRC, add the following content.

?

123	# Spark Envexport SPARK_HOME=/opt/spark-1.6.1-bin-hadoop2.6export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

Run the source. BASHRC to make the environment variable effective.

(4) Spark configuration

Go to the Conf directory under the Spark installation directory and copy the Spark-env.sh.template to spark-env.sh.

?

1	cp spark-env.sh.template spark-env.sh

Edit spark-env.sh, where you add the following configuration information:

?

12345

export SCALA_HOME=/opt/scala-2.11.8export JAVA_HOME=/opt/java/jdk1.7.0_80export SPARK_MASTER_IP=192.168.109.137export SPARK_WORKER_MEMORY=1gexport HADOOP_CONF_DIR=/opt/hadoop-2.6.4/etc/hadoop

JAVA_HOME Specifies the Java installation directory;
Scala_home Specify the Scala installation directory;
SPARK_MASTER_IP Specifies the IP address of the Spark cluster MASTER node;
SPARK_WORKER_MEMORY Specifies the maximum memory size that the WORKER node can allocate to executors;
HADOOP_CONF_DIR Specifies the Hadoop cluster configuration file directory.

Copy the slaves.template to slaves and edit its contents as:

?

123	masterslave01slave02

That is, Master is both the master node and the Worker node.

2. Slave machine

SLAVE01 and SLAVE02 are installed by referring to the master machine installation steps.

Iv. start the Spark cluster 1, start the Hadoop cluster

The start of a Hadoop cluster can be found in a previous article on the Hadoop 2.6.4 distributed cluster environment, which is not described here. Once started, you can use the JPS command to view process information on master, SLAVE01, SLAVE02, respectively.

2. Start the Spark cluster

(1) Start Master node

Run start-master.sh with the following results:

You can see that there is a new process master on master.

(2) Start all Worker nodes

Run the start-slaves.sh and run the following results:

Using the JPS command on Master, SLAVE01, and SLAVE02, you can see that a Worker process has been started

(3) The browser views the Spark cluster information.

Access: http://master:8080, such as:

(4) Using Spark-shell

Running Spark-shell, you can enter the shell console of Spark, as follows:

(5) Browser access Sparkui

Access http://master:4040, such as:

You can view information such as environment variables, jobs, executor, and so on from Sparkui.

At this point, the entire Spark distributed cluster is built to end here.

V. Stop the Spark cluster

1. Stop Master node

Run stop-master.sh to stop the master node.

To view the current Java process using the JPS command

You can see that the Master process has stopped.

2. Stop the Worker node

Run stop-slaves.sh to stop all Worker nodes

Use the JPS command to view process information on master, SLAVE01, Slave02:

As you can see, the Worker process is stopped and then the Hadoop cluster is stopped.

VI. Legacy Issues

The set of Scala_home does not take effect, and Spark starts with the self-brought SCALA 2.10.5 version.

Spark 1.6.1 Distributed cluster Environment setup