Apache Spark Source code reading 15 -- Fault Tolerance Analysis in standalone deployment mode

You are welcome to reprint it. Please indicate the source, huichiro.

Summary

This article makes a more detailed analysis of fault tolerance issues in standalone deployment mode, mainly answering the main nodes contained in standalone deployment mode, and how the system handles problems in a certain type of nodes.

Nodes deployed on Standalone

The introduction to spark involves a lot of RDD concepts. However, there are not many questions about how RDD runs and how it maps to processes and threads.

In the actual production environment, spark always runs in the cluster mode. standalone deployment mode is the most streamlined in all cluster mode, and mesos and yarn are also used, it takes more time to understand the internal operation mechanism.

Composition of standalone Cluster

A standalone cluster consists of three nodes of different levels, which are

• MasterThe master node can be analogous to the chairman of the board or the master. In the whole cluster, only one master node is active at most.
• WorkerWorking node. This is the manager, which is the sub-master. There can be multiple workers in the entire cluster. If the worker is zero, nothing can be done.
• ExecutorThe worker is directly controlled by the worker. One worker can start multiple executors, and the number of executors is limited by the number of CPU cores in the machine.

These three different types of nodes run in their own JVM processes..

Driver Application

The application submitted to the standalone cluster is called driver applicaton.

Standalone cluster startup and task submission

 

Summarizes messages between nodes when the standalone cluster is started normally and when the application is submitted. The following describes the process of cluster startup and application submission in detail.

Cluster Startup Process

The normal startup process is described as follows:

Step 1: Start the master

$SPARK_HOME/sbin/start-master.sh

Step 2: Start worker

./bin/spark-class org.apache.spark.deploy.worker.Worker spark://localhost:7077

After the worker is started, two tasks are performed.

1. Register yourself to master, registerworker
2. Send heartbeat messages to the master periodically

Task submission process step 1: Submit Application

Run the following command to start spark-shell:

MASTER=spark://127.0.0.1:7077 $SPARK_HOME/bin/spark-shell

When spark-shell is run, a registerapplication request is sent to the master.

Log location:Logs generated by the master operation are stored in$ Spark_home/logsDirectory

Step 2: After the master processes the registerapplication request

After receiving the registerapplication request, mastet will process it as follows:

1. If a worker has been registered, send the launchexecutor command to the corresponding worker.
2. If no, nothing will happen.

Step 3: Start executor

After the worker receives the launchexecutor command, it starts the executor process.

Step 4: Register executor

The started executor process registers itself to schedulerbackend In the Driver Based on the input parameters at startup.

Log location: The Running log of executor is in$ Spark_home/workDirectory

Step 5: run the task

After schedulerbackend receives the registration message of executor, it splits the submitted spark job into multiple specific tasks, and then disperses these tasks to various executors for real operation through the launchtask command.

If no executor is registered to schedulerbackend when runjob is called, what is the corresponding processing logic?

1. Schedulerbackend stores tasks in TASKMANAGER.
2. Once an executor is registered, submit the tasks that have not been run managed by taskmanager to the executor.
3. If multiple jobs are in the pending state, the default scheduling policy is FIFO, that is, the first submitted job runs first.

Test procedure

1. Start master
2. Start spark-shell
3. Run SC. textfile ("readme. md"). Count
4. Start worker
5. Note the log messages printed in spark-shell after the worker is started.

Job execution ended

When the task ends, the corresponding executor is stopped.

The following test can be performed:

1. Stop spark-shell
2. You can use PS-Ef | grep-I Java to view the Java Process.CoarsegrainedexecutorbackendProcess exited

Summary

The order between the control message primitives above can be seen.

1. The master and worker processes must be started explicitly.
2. Executor is implicitly taken by worker.
3. Cluster startup sequence
   1. The master must be started before other nodes.
   2. It doesn't matter which worker and driver start first
   3. However, the jobs submitted by the driver can be actually executed only after the corresponding worker is registered to the master.

Exception Scenario Analysis

The above shows the message distribution details of each node under normal circumstances. If a problem occurs on some nodes in the cluster during running, can the whole cluster still process tasks in the application normally?

Exception Analysis 1: worker exits unexpectedly

During spark running, the common problem is that the worker exits abnormally. When the worker exits, what stories will happen to the entire cluster? See the following description.

1. The worker exits unexpectedly. For example, the worker is consciously killed by the kill command.
2. Before the worker exits, all executors managed by the worker will be killed.
3. The worker needs to regularly improve heartbeat messages to the master. Now the worker process has been completed and there is a heartbeat message, so the master will realize that a "Rudder" has left in the timeout process.
4. The master is very sad, and the sad master reports the situation to the corresponding driver.
5. The driver confirmed that the executor assigned to him had unfortunately left. One was the notification sent by the master, and the other was not received by the driver within the specified time, the driver will remove the registered executor.

Consequence analysis

Impact of abnormal worker exit

1. When the executor exits, the submitted task cannot end normally and will be submitted and run again.
2. If all workers exit abnormally, the entire cluster is unavailable.
3. A program is required to restart the worker process, for exampleSupervisordOrRunit

Test procedure

• Start master
• Start worker
• Start spark-shell
• Manually kill the Worker Process
• Use JPs or PS-Ef | grep-I Java to view the started Java Process

Abnormal exit code processing

Defined in executorrunner. Scala's start Function

def start() {    workerThread = new Thread("ExecutorRunner for " + fullId) {      override def run() { fetchAndRunExecutor() }    }    workerThread.start()    // Shutdown hook that kills actors on shutdown.    shutdownHook = new Thread() {      override def run() {        killProcess(Some("Worker shutting down"))      }    }    Runtime.getRuntime.addShutdownHook(shutdownHook)  }

Killprocess is the process of stopping the corresponding coarsegrainedexecutorbackend.

When a worker is stopped, it is necessary to stop its executor. Is this like the means of Song Jiang in the middle of the Song Dynasty, and Li Yu is so desperate to lose his life.

Summary

It should be pointed out that when a worker starts executor, it is done through executorrunner. executorrunner is an independent thread and has a one-to-one relationship with executor, which is very important. Executor is running as an independent process, but it is closely monitored by executorrunner.

Exception Analysis 2: Executor exits unexpectedly

As the bottom-layer employee in standalone cluster deployment mode, what are the consequences of an exceptional exit?

1. Executor exits abnormally. executorrunner notices the exception and reports the exception to the master through executorstatechanged.
2. After the master receives the notification, it is very unhappy. However, some younger brother is about to run, and the worker of the executor is required to start again.
3. Worker receives the launchexecutor command and starts executor again.

As a bottom-layer employee, it is impossible to pick a child easily. "You cannot help yourself when you are in the rivers and lakes.

Test procedure

• Start master
• Start worker
• Start spark-shell
• Manually kill coarsegrainedexecutorbackend

Fetchandrunexecutor

FetchandrunexecutorStarts the executor and monitors its running status. The specific code logic is as follows:

def fetchAndRunExecutor() {    try {      // Create the executor‘s working directory      val executorDir = new File(workDir, appId + "/" + execId)      if (!executorDir.mkdirs()) {        throw new IOException("Failed to create directory " + executorDir)      }      // Launch the process      val command = getCommandSeq      logInfo("Launch command: " + command.mkString("\"", "\" \"", "\""))      val builder = new ProcessBuilder(command: _*).directory(executorDir)      val env = builder.environment()      for ((key, value)  {        logInfo("Runner thread for executor " + fullId + " interrupted")        state = ExecutorState.KILLED        killProcess(None)      }      case e: Exception => {        logError("Error running executor", e)        state = ExecutorState.FAILED        killProcess(Some(e.toString))      }    }  }

Exception Analysis 3: The Master exits unexpectedly.

 

The worker and executor quit abnormally. The last case is left. What should I do if the master fails?

What will happen if the Lead Brother is absent?

• The worker has no objects to report, that is, if the executor runs again, the worker will not start the executor.
• Unable to submit new tasks to the Cluster
• Even if the old task is finished, the occupied resources cannot be cleared, because the command for clearing resources is issued by the master.

How do you know the consequences are serious? Don't look at the boss's daily work. If you really don't want to be there, you can't just rely on your younger siblings.

Solution to master spof

So how can we solve the single point of failure (spof) Problem of the master?

You just need to add another master, two bosses. If the two bosses both have Command Permissions, the results will be disastrous. Set up a deputy employee. When the current Current current official job fails, the deputy takes over. That is, there is only one active master at the same time.

Note: Yes. How can this problem be implemented? Use the electleader function of zookeeper as follows:

Configuration details

How to Set up a zookeeper cluster is no longer nonsense here. If you have time, you can set up the cluster again or refer to the zookeeper cluster installation steps mentioned in the storm series.

If the zookeeper cluster has been set successfully, how can I start the nodes in the standalone cluster? What's special?

Conf/spark-env.sh

In the conf/spark-env.sh, add the following options for spark_daemon_java_opts

System Property	Meaning
Spark. Deploy. recoverymode	Set to zookeeper to enable standby master recovery mode (default: none ).
Spark. Deploy. zookeeper. url	The Zookeeper cluster URL (e.g., 192.168.1.100: 2181,192.168 .1.101: 2181 ).
Spark. Deploy. zookeeper. dir	The directory in zookeeper to store recovery state (default:/spark ).

Example of setting spark_daemon_java_opts

SPARK_DAEMON_JAVA_OPTS="$SPARK_DAEMON_JAVA_OPTS -Dspark.deploy.recoveryMode=ZOOKEEPER"

Application startup

When the application is running, multiple master addresses are specified and separated by commas, as shown below:

MASTER=spark://192.168.100.101:7077,spark://192.168.100.102:7077 bin/spark-shell

Summary

Fault Tolerance Analysis in standalone cluster deployment gives us a further understanding of Spark's task distribution process. The previous chapter hurried through the knowledge points involved in spark from the whole, and the analysis was not deep enough and not detailed enough.

This article attempts to provide an in-depth analysis of a specific problem. When analyzing the framework, we can"Big Open and big cooperation, shuke can take a horse, counted as black"In detail analysis, we need to"Airtight, analysis, progressive".