How the spark cluster works under HA (dt Big Data Dream Factory)

Spark High Availability ha combat

How the Spark cluster works

Resources mainly refer to memory, CPU

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>

If it is a single point, if Master fails, the cluster cannot work externally

Spark does ha through zookeeper, generally doing HA is an active level, standby

Active is the current job

Standby is ready to switch to the active level after the active hangs

Used to be 2 machines, one active, one standby

It is now generally 3 machines, one active, two standby, or even more than 3 units

What zookeeper contains: All Woker, Driver, application, when active hangs, zookeeper selects a standby as leader and then restores Woker, Driver, Application, after these are completed, leader will be programmed covering, in order to program the active, only when it becomes active to continue to provide services, normal job submission

Will a cluster run as a result of master switching affecting the operation of the business program? Not at coarse granularity, because the program has already applied resources to master before it runs, and then the job is just the interaction between the worker and the Excutor, and master doesn't matter, this is the resource that was allocated before the switch. This is a coarse-grained case. Fine-grained.

Fine-grained drawbacks, task initiation is very slow. Normal large-size data processing, the actual situation, is generally coarse granularity.

Get started:

========== Download zookeeper============

http://zookeeper.apache.org/

Download it and unzip it to/usr/local/, then configure the environment variable.

Export zookeeper_home=/usr/local/zookeeper-3.4.6

Export path= $PATH: ${zookeeper_home}/bin

Zookeeper is installed separately.

Because it's distributed, so the zookeeper is going to be put on more than one machine.

We put it in Worker1, Worker2.

Go to zookeeper, create data and logs two directories

[Email protected]:/usr/local/zookeeper-3.4.6# mkdir Data

[Email protected]:/usr/local/zookeeper-3.4.6# mkdir Logs

[Email protected]:/usr/local/zookeeper-3.4.6/conf# VI zoo_sample.cfg

DataDir This must be modified, or the data will be deleted after reboot

[Email protected]:/usr/local/zookeeper-3.4.6/conf# cp zoo_sample.cfg ZOO.CFG

[Email protected]:/usr/local/zookeeper-3.4.6/conf# VI zoo.cfg

Modify (Make a cluster of 3 machines)

Datadir=/usr/local/zookeeper-3.4.6/data

Datalogdir=/usr/local/zookeeper-3.4.6/logs

server.0=master:2888:3888

server.1=worker1:2888:3888

server.2=worker2:2888:3888

[Email protected]:/usr/local/zookeeper-3.4.6/conf# CD. /data/

Number The Machine

[Email protected]:/usr/local/zookeeper-3.4.6/data# Echo 0>myid

[Email protected]:/usr/local/zookeeper-3.4.6/data# Echo 0>>myid

[Email protected]:/usr/local/zookeeper-3.4.6/data# ls

myID

[Email protected]:/usr/local/zookeeper-3.4.6/data# cat myID

[Email protected]:/usr/local/zookeeper-3.4.6/data# VI myID

[Email protected]:/usr/local/zookeeper-3.4.6/data# cat myID

0

[Email protected]:/usr/local/zookeeper-3.4.6/data#

By this time, a machine has been configured.

Zookeeper after the cluster is configured, when spark restarts, you can synchronize the information of the last cluster. Otherwise, spark restarts and it's all over again.

[Email protected]:/usr/local# scp-r/zookeeper-3.4.6 [email protected]:/usr/local

[Email protected]:/usr/local# scp-r/zookeeper-3.4.6 [email protected]:/usr/local

Then go in separately Worker1 and Worker2 change myID to 1 and 2

[Email protected]:/usr/local/zookeeper-3.4.6# CD Bin

[Email protected]:/usr/local/zookeeper-3.4.6/bin# ls

README.txt Zkcli.cmd Zkenv.cmd Zkserver.cmd

zkcleanup.sh zkcli.sh zkenv.sh zkserver.sh

[Email protected]:/usr/local/zookeeper-3.4.6/bin# CD Bin

Bash:cd:bin:No such file or directory

[Email protected]:/usr/local/zookeeper-3.4.6/bin# zkserver.sh Start

Then three machines are activated separately.

The next step is to have spark support zookeeper ha

Configure in Spark-env.sh

[Email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/conf# VI spark-env.sh

The state of the entire cluster is maintained and restored through zookeeper, and the status information is

Export spark_daemon_java_opts= "-dspark.deploy.recoverymode=zookeeper-dspark.deploy.zookeeper.url=master : 2181,worker1:2181, worker2:2181 -dspark.deploy.zookeeper.dir=/spark "

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>

The cluster has been configured, so also note

#export Spark_master_ip=master

[Email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/conf# SCP spark-env.sh [email protected]:/usr/local/ spark-1.6.0-bin-hadoop2.6/conf/spark-env.sh

spark-env.sh 100% 0.5kb/s 00:00

[Email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/conf# SCP spark-env.sh [email protected]:/usr/local/ spark-1.6.0-bin-hadoop2.6/conf/spark-env.sh

spark-env.sh 100% 0.5kb/s 00:00

And then it's the boot spark,./start-all.sh.

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>

Why is there master process in master, Worker1, 2?

[Email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin# CD. /conf/

[Email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/conf# cat Slaves

Master

Worker1

Worker2

Since we have only one master, we must start Master on Worker1 and 2

[Email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/bin# cd $SPARK _home/sbin

[Email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin#./start-master.sh

Starting Org.apache.spark.deploy.master.Master, logging to/usr/local/spark-1.6.0-bin-hadoop2.6/logs/ Spark-root-org.apache.spark.deploy.master.master-1-worker1.out

[Email protected]:/usr/local/zookeeper-3.4.6/bin# cd $SPARK _home/sbin

[Email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/sbin#./start-master.sh

Starting Org.apache.spark.deploy.master.Master, logging to/usr/local/spark-1.6.0-bin-hadoop2.6/logs/ Spark-root-org.apache.spark.deploy.master.master-1-worker2.out

Http://master:8080/look at the situation

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>

http://worker1:8080/

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>

No worker, standby state, full spare tire status.

http://worker2:8080/

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>

No worker, standby state, full spare tire status.

Use Spark-shell to experiment with HA

[Email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/conf# cd $SPARK _home/bin

[Email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/bin#./spark-shell--master Spark://master:7077,worker1 : 7077,worker2:7077

Startup logs are registered:

16/02/04 21:12:29 INFO Client. Appclient$clientendpoint:connecting to Master spark://master:7077 ...

16/02/04 21:12:29 INFO Client. Appclient$clientendpoint:connecting to Master spark://worker1:7077 ...

16/02/04 21:12:29 INFO Client. Appclient$clientendpoint:connecting to Master spark://worker2:7077 ...

But it actually only interacts with the active machine.

Come on CD $SPARK _home/sbin

./stop-master.sh

See there's a movement in Spark-shell.

scala> 16/02/04 21:17:11 WARN client. Appclient$clientendpoint:connection to master:7077 failed; Waiting for Master to reconnect ...

16/02/04 21:17:11 WARN cluster. Sparkdeployschedulerbackend:disconnected from Spark cluster! Waiting for reconnection ...

16/02/04 21:17:11 WARN Client. Appclient$clientendpoint:connection to master:7077 failed; Waiting for Master to reconnect ...

16/02/04 21:17:44 INFO Client. Appclient$clientendpoint:master has changed, new Master was at spark://worker1:7077

Worker1 becomes active for half a minute, not immediately, programming active depending on the size of the cluster

Http://master:8080/'s not even up.

http://worker1:8080/

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>

http://worker2:8080/, or just like that.

Run the PI calculation

./spark-submit --class org.apache.spark.examples.SparkPi --master spark://master:7077,worker1:7077,worker2:7077. /lib/spark-Examples-1.6.0-hadoop2.6.0.jar100

Then start master of Master machine

At this point the master machine becomes the standby mode

http://master:8080/

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>

General production Environment 100, 200G memory

Liaoliang Teacher's card:

China Spark first person

Sina Weibo: Http://weibo.com/ilovepains

Public Number: Dt_spark

Blog: http://blog.sina.com.cn/ilovepains

Mobile: 18610086859

qq:1740415547

Email: [Email protected]

This article from "a Flower proud Cold" blog, declined reprint!

How the spark cluster works under HA (dt Big Data Dream Factory)