Spark Standalone HA

Configuring Spark Standalone HA

Host: Node1,node2,node3

Master:node1,node2

Slave:node2,node3

To modify a configuration file:

node1,node3:spark-env.sh

Export spark_master_ip=node1export spark_master_port=7077export Spark_worker_cores= 1 export Spark_worker_instances=1export spark_worker_memory=1024mexport Spark_local_ DIRS=/data/spark/datadirexport spark_daemon_java_opts="-dspark.deploy.recoverymode= Zookeeper-dspark.deploy.zookeeper.url=node1:2181,node2:2181,node3:2181-dspark.deploy.zookeeper.dir=/sparkha "

node2:spark-env.sh

Export Spark_master_ip=node2

The difference between Node2 and Node1 is only here

Startup script:

Zookeeper has been started, there is no description of zookeeper configuration and startup.

Spark startup script

Node1:

/sbin/start-all.sh

Node2:

/sbin/start-master.sh

Test ha

Shut down Node1 Master.

/sbin/stop-master.sh

Access the Master of Node2

Did not stop Node1 's master, Access Node2

After you have stopped Node1 master, visit node2

Spark Standalone HA