Two of the most common fault-tolerant scenarios of Hadoop MapReduce

This article will analyze two common fault-tolerant scenarios for Hadoop MapReduce (including MRv1 and MRv2), the first of which is that a task is blocked, the resource is not released for a long time, and how to handle it? The other is that the map of the job is http://www.aliyun.com/ Zixun/aggregation/17034.html "When the >task is complete, how do I handle a map task when it is run, or if a map task result is damaged by a disk?"

The first scenario: a task is blocked, long time to occupy resources do not release, how to deal with?

This scenario is usually caused by software bugs, data specificity, and so on, blocking the program and halting the task. In the eyes of the outside world, the task seems to be blocked. This kind of thing often occurs, because the task takes up the resources for a long time but does not use (if does not take certain means, may never be used, causes "the resource leakage"), will cause the resource utilization rate to reduce, to the system disadvantage, then, Hadoop MapReduce encounters this situation how to handle?

On Tasktracker, each task periodically reports the new progress to the Tasktracker (if the progress is constant, does not report) and is further reported to Jobtracker by Tasktracker. When a task is blocked, its progress will stagnate, at which point the task does not report progress to tasktracker, so that it must reach the time-out limit, Tasktracker will kill the task and report the task status (killed) to Jobtracker, This triggers jobtracker to reschedule the task.

In a real-world scenario, there are normal jobs whose tasks may not be read or output for long periods of time, such as reading a map task for a database or a task that needs to connect to other external systems, for such applications, when writing mapper or reducer, An additional thread should be started to periodically report the heartbeat to Tasktracker via the reporter component (just tell Tasktracker I'm alive and Don't Kill me).

Second scenario: When the map task for a job is complete, how do I handle a map task when it is run, or the map result is damaged by the disk?

This scenario is more complex and needs to be discussed separately.

If the node hangs, Jobtracker knows that Tasktracker is dead through the heartbeat mechanism, and it will reschedule the task that was running and the map task that was already running in the running job.

If the node is not hanging, only the disk that holds the map task result is corrupted, in two cases:

(1) All reduce tasks have completed the shuffle phase

(2) A partial reduce task does not complete the shuffle phase and needs to read the Map task task

In the first case, if all the reduce tasks go well, there is no need to do any processing on the map task that is already running, and if some reduce task fails after a period of time, the process is the same as the second.

In the second case, when the reduce task remotely reads the result of the map task that is already running (but the result is corrupted), it attempts to read several times, and if the number of attempts exceeds a certain upper bound, the Tasktracker is told by RPC that the map The Task result is corrupted, and Tasktracker further evaluates the build results by rescheduling the map task when the message is received by RPC to tell Jobtracker,jobtracker.

It should be emphasized that in the current implementation of the Hadoop MapReduce, the time interval for the Reduce task to retry reading the map task result is exponentially incremented, and the formula is 10000*1.3^nofailedfetches, Where the range of nofailedfetches values is max{10, NUMMAPS/30}, that is, if the map task number is 300, you need to try 10 times to find that the map task result is corrupted, and the attempt interval is 10s,13s,21s, 28s,37s,48s,62s,81s and 106s, which takes a very long time to discover, and the more Map tasks, the slower the discovery, the place usually needs to be tuned because the more tasks there are, the more likely this problem will occur.

In MapReduce V2.0, All Tasks (Map task and reduce task) interact directly with Mrappmaster and do not need to pass through a middle tier similar to Tasktracker, which is similar to this process, and is not discussed here. Specific reading book "Hadoop Technology Insider: In-depth analysis of yarn architecture design and implementation of the principle of" the 8th Chapter off-line computing Framework MapReduce.