Spark-1.0.0 standalone distributed installation tutorial

Spark currently supports multiple distributed deployment modes: 1. standalone deploy mode; 2. Amazon EC2; 3. Apache mesos; 4. hadoop yarn. The first method is to deploy spark separately without the dependency Resource Manager. The other three methods need to deploy spark on the corresponding resource manager.

In addition to multiple deployment methods, a newer version of spark supports multiple hadoop platforms, for example, hadoop 1 (hdp1, cdh3), cdh4, and hadoop 2 (hdp2, cdh5 ). Currently, when cloudera cdh5 is installed with cm, you can directly select the spark service for installation.

The latest version of spark is 1.0.0.

Let's take version 1.0.0 to see how to install a spark distributed cluster:

I,Spark 1.0.0 requires JDK 1.6 or later. JDK 1.6.0 _ 31 is used here;

II,Spark 1.0.0 requires Scala 2.10 or later. Scala 2.10.3 is used here;

III,Download the appropriate bin package from the https://spark.apache.org/downloads.html for installation, here we select the cdh4 version of The spark-1.0.0-bin-cdh4.tgz; download to tongjihadoop165;

IV,Decompress the bin package: tar-zxf spark-1.0.0-bin-cdh4.tgz;

V,Rename: MV spark-1.0.0-bin-cdh4 spark-1.0.0-cdh4;

VI,CDSpark-1.0.0-cdh4;

MV./CONF/spark-env.sh.template./CONF/spark-env.sh

VII,Vi./CONF/spark-env.sh Add the following:

Export scala_home =/usr/lib/scala-2.10.3

Export java_home =/usr/Java/jdk1.6.0 _ 31

Export spark_master_ip = 10.32.21.165

Export spark_worker_instances = 3

Export spark _ master_port = 8070

Export spark_master_webui_port = 8090

Export spark _ worker_port = 8092

Export spark _ worker_memory = 5000 m

 

Spark_master_ip refers to the Master Ip address; spark_master_port refers to the master port; spark_master_webui_port refers to the Web UI port number for viewing the cluster running status; spark_worker_port refers to the port number of each worker; spark_worker_memory: configure the running memory of each worker.

8,Vi./CONF/slaves each line has a worker host name. The content is as follows:

10.32.21.165

10.32.21.166

10.32.21.167

IX,(Optional) set the spark_home environment variable and add spark_home/bin to path:

VI/etc/profile:

Export spark_home =/usr/lib/spark-1.0.0-cdh4

Export Path = $ spark_home/bin: $ path

10,Copy spark on tongjihadoop165 to tongjihadoop166 and tongjihadoop167:

Sudo SCP-R [e-mail protected]:/usr/lib/spark-1.0.0-cdh4/usr/lib

You can also remotely copy the file and modify the environment variable file/etc/profile when installing scala. Do not forget source after the change.

XI,Run./sbin/start-all.sh to start the spark cluster;

If the start-all method cannot start the relevant processes normally, you can view the related error information in the $ spark_home/logs directory. In fact, you can also start related processes independently like hadoop and run the following command on the master node:

Run:./sbin/start-master.sh on the master

Run:./sbin/start-slave.sh 3 spark: // 10.32.21.165: 8070 -- webui-port 8090 on worker

12,Check whether the process is started and run the JPS command to view the worker process or master process. Then you can view http: // tongjihadoop165: 8090/on the Web UI to view all the work nodes, as well as their CPU count and memory information.

XIII,Run demo in Local Mode

For example:./bin/run-example sparklr 2 local or./bin/run-example sparkpi 2 Local

In the two examples, the former is linear regression and iterative calculation; the latter is the calculation of the circumference rate.

14th,Start Interactive Mode:./bin/spark-shell -- master spark: // 10.32.21.165: 8070

This shell is a modified Scala shell. Opening such a shell will show a running application in the Web UI, such:

At the bottom, applications are completed. The workers list is the node list of the cluster.

We can perform some calculations on the data on HDFS in the opened shell, and input the following in the shell:

A. Val file = SC. textfile ("HDFS: // 10.32.21.165: 8020/1639. Sta") # This is the file in HDFS loading.

B. file. Map (_. Size). Reduce (_ + _) # This is the number of characters in the computing file.

Running status, such:

 

The final result is 346658513 characters long. The speed is less than 3 s.

Or execute Val COUNT = file in stage B. flatmap (line => line. split ("\ t ")). map (WORD => (word, 1 )). performancebykey (_ + _) and count. saveastextfile ("HDFS: // 10.32.21.165: 8020/spark") stores the computing results in the/spark directory on HDFS.

You can also run./bin/spark-shell -- master local [2] to start a local shell. [2] can specify the number of threads. The default value is 1.

Run exit to exit shell.

15th,Run./sbin/stop-all.sh to stop the spark Cluster

It can also be terminated through a stop script of a separate process.

 

 

Note: three machinesSparkThe directory must be consistent becauseMasterWill log onWorkerRun the command,MasterThinkWorkerOfSparkThe path is the same as your own.

 

Refer:

1. http://www.linuxidc.com/Linux/2014-06/103210p2.htm

2. http://spark.apache.org/docs/latest/

3. http://blog.csdn.net/myrainblues/article/details/22084445