[Interactive Q & A sharing] The 18th issue won the big data era of cloud computing, spark Asia Pacific Research Institute public welfare Lecture Hall (change)

 

"Winning the cloud computing Big Data era"

Spark Asia Pacific Research Institute Stage 1 Public Welfare lecture hall [Stage 1 interactive Q & A sharing]

 

Q1: Is the master and driver the same thing?

• The two are not the same. In standalone mode, the master node is used for cluster resource management and scheduling, while the driver is used to command executors on the worker to process tasks in multiline mode;

• The master node is located on the Management node of the cluster. Generally, it is on the same node as namenode;

• Generally, the driver is located on the client. The client generally does not belong to the cluster, but is in the same network environment as the cluster, because the driver in the client needs to frequently interact with the executor in the cluster;

 

Q2: How to Select between standalone and Yarn

• Standalone and yarn are both systems used for resource management. standalone is a lightweight resource management and allocation method specially designed for spark, and yarn is a general resource management framework for big data, it can be used not only to manage spark vertex resource allocation, but also to manage and allocate resources for other computing platforms of yarn;

• If multiple computing frameworks such as spark, mapreduce, and mahout coexist in the production system, we recommend that you use yarn or mesos for unified resource management and scheduling. If you only use spark, standalone is enough, and yarn consumes resources;

Q3: How does spark's Ha handle it?

• For Master ha, in standalone mode, the worker node is automatically ha. For Master ha, Zookeeper is generally used;

• Utilizing zookeeper to provide leader election and some state storage, You can launch multiple masters in your cluster connected to the same zookeeper instance. one will be elected "leader" and the others will remain in standby mode. if the current leader dies, another master will be elected, recover the old master's state, and then resume scheduling. the entire recovery process (from the time the first leader goes down) shocould take between 1 and 2 minutes. note that this delay only affects scheduling new applications-applications that were already running during master failover are unaffected;

• For yarn and mesos modes, ResourceManager generally uses zookeeper for HA;

[Interactive Q & A sharing] The 18th issue won the big data era of cloud computing, spark Asia Pacific Research Institute public welfare Lecture Hall (change)