[Interactive Q & A sharing] Stage 1 wins the public welfare lecture hall of spark Asia Pacific Research Institute in the cloud computing Big Data age

 

"Winning the cloud computing Big Data era"

Spark Asia Pacific Research Institute Stage 1 Public Welfare lecture hall [Stage 1 interactive Q & A sharing]

Q1: Is the master and driver the same thing?

• The two are not the same. In standalone mode, the master node is used for cluster resource management and scheduling, while the driver is used to command executors on the worker to process tasks in multiline mode;

• The master node is located on the Management node of the cluster. Generally, it is on the same node as namenode;

• Generally, the driver is located on the client. The client generally does not belong to the cluster, but is in the same network environment as the cluster, because the driver in the client needs to frequently interact with the executor in the cluster;

 

Q2: How to Select between standalone and Yarn

• Standalone and yarn are both systems used for resource management. standalone is a lightweight resource management and allocation method specially designed for spark, and yarn is a general resource management framework for big data, it can be used not only to manage spark vertex resource allocation, but also to manage and allocate resources for other computing platforms of yarn;

• If multiple computing frameworks such as spark, mapreduce, and mahout coexist in the production system, we recommend that you use yarn or mesos for unified resource management and scheduling. If you only use spark, standalone is enough, and yarn consumes resources;

Q3: How does spark's Ha handle it?

• For Master ha, in standalone mode, the worker node is automatically ha. For Master ha, Zookeeper is generally used;

• Utilizing zookeeper to provide leader election and some statestorage, You can launch multiple masters in your cluster connected to the samezookeeper instance. one will be elected "leader" and the others will remain instandby mode. if the current leader dies, another master will be elected, recover the old master's state, and then resume scheduling. the entire recoveryprocess (from the time the first leader goes down) shocould take between 1and 2 minutes. note that this delay only affectsscheduling new applications-applications that were already runningduring master failover are unaffected;

• For yarn and mesos modes, ResourceManager generally uses zookeeper for HA;

This article is from the spark Asia Pacific Research Institute blog, please be sure to keep this source http://rockyspark.blog.51cto.com/2229525/1570381

[Interactive Q & A sharing] Stage 1 wins the public welfare lecture hall of spark Asia Pacific Research Institute in the cloud computing Big Data age