Fault Tolerance of Storm

Fault Tolerance of Storm

This article describes the design details of Storm Fault Tolerance: How the Worker, node, Nimbus, or Supervisor is Fault tolerant, and whether Nimbus has a single point of failure.

This blog is about the translation of the Fault-Tolerance article on the Storm official website. I have been paying attention to Storm-related technologies and found that although this article is rarely written on the official website, it describes the main design details of Storm fault tolerance (apart from ensuring data processing, it has been explained in detail in the Guaranteeing message processing article, so this article only gives a link ).

What happens when a Worker crashes?

When a Worker fails, the Supervisor restarts it. If the Worker fails continuously at startup and cannot let Nimbus observe its heartbeat, Nimbus will reassign the Worker to another machine.

What happens when a node fails?

Tasks assigned to this machine will time out and Nimbus will reassign these tasks to other machines.

What happens when the Nimbus or Supervisor daemon process goes down?

The Nimbus and Supervisor daemon processes are designed to be fast-failed (self-destructed whenever any exception occurs) and stateless (all statuses are saved on Zookeeper or disk ). As described in Setting up a Storm cluster, the Nimbus and Supervior daemon processes must run under monitoring, such as using daemontools or monit tools. So if the Nimbus or Supervisor daemon process goes down, it can be restarted like nothing happens.

It is very important that no Worker process will be affected by the failure of Nimbus or Supervisor. This is opposite to Hadoop. If JobTracker fails in Hadoop, all running jobs will be lost.

Does Nimbus have a single point of failure?

When you lose the Nimbus node, the Worker can continue to work. In addition, the Supervisor can continue to restart the crashed Worker. However, without the Nimbus node, the Worker cannot be reassigned to other machines as needed. (For example, you have lost a Woker machine ).

So the answer is that Nimbus has a single point of failure. In practice, this is not a big problem. The failure of the Nimbus deamon process will not cause any disaster. In the future, we plan to design Nimbus into high availability.

How does Storm ensure data processing?

Storm provides some mechanisms to ensure that data processing can be performed correctly even if the node fails or the message is lost. For more information, see Guaranteeing message processing.

Storm Process Communication Mechanism Analysis

Apache Storm History and Lessons

For details about Apache Storm, click here
Apache Storm: click here

This article permanently updates the link address: