Summary of Storm distributed real-time flow computing framework related technologies

Summary of Storm distributed real-time flow computing framework related technologies

As an open-source distributed real-time streaming computing framework, Storm's internal implementations use a number of common techniques, which are an overview of these technologies and their role in storm. Based on this, follow-up further insights into Storm's internal implementation details.

1. Zookeeper Cluster
Zookeeper is a reliable coordinated service system for large distributed systems that uses a data model similar to the UNIX File system tree hierarchy (e.g.,/zoo/a,/zoo/b), where a small amount of data can be stored in the node (<1m, when the node stores a large amount of data , synchronization problems may occur in the actual application).

Zookeepr's high-reliability service is implemented by configuring the zookeeper cluster, and the official recommendation is that the zookeeper cluster contain at least 3 nodes, one copy of the data on each node, and a node that can be re-selected as the primary node after the primary node is hung off. As long as more than half of the nodes in the cluster survive, the cluster can provide services externally.

Zookeeper supported operation types are:
(1) Create tree node Znode;　　
(2) Delete tree node Znode;


Span style= "font-size:15px;" > (5) Support instantaneous node (ephemeral), the node is automatically deleted after the process of creating the node exits.

storm is mainly used for distributed coordination of each node in the storm cluster, with the following functions:
(1) Store the topology task information provided by the client (that is, the storm real-time process order), Nimbus is responsible for writing task assignment Information Zookeeper,supervisor Read task assignment information from zookeeper;
(2) Stores supervisor and worker Heartbeat (package The Nimbus can monitor the status of the entire cluster, thus restarting some of the dead workers;
(3) stores all state information and configuration information for the entire cluster.

For The detailed directory structure of storm in zookeeper, please visit: The directory structure in the zookeeper of Twitter Storm source code Analysis

2. Thrift Service Framework
Thrift is a cross-language extensible Service Framework that defines RPC interfaces and data types through an intermediate language (IDL, Interface Definition Language), and then generates a seamless cross-programming language for RPC client and server communication through a compiler.

　　Scenarios for thrift in storm:
(1) The client submits topology task to Nimbus;
(2) Supervisor Download topology task (code and serialization file) from Nimbus;
(3) The Storm UI gets statistics for topology runs from Nimbus.

3. ZEROMQ Message Queuing --netty
ZEROMQ is a message-based embedded network programming library that can be used as a concurrent framework to connect multiple applications, support n-to-n connections, multiple working modes (request-reply,publish-subscribe,pipeline, etc.), support multiple languages, ZEROMQ makes writing high-performance Web applications extremely simple.

　　ZEROMQ scenario in Storm: the transmission of a tuple message between spout and Bolt, bolt, and bolt.

4. Java serialization
　　Java serialization technology can be implemented to save Java objects as a binary file , and the deserialization process can be in another Java process to restore this binary file to Java objects, the disadvantage is that the version changes are not well resolved.

　　Scenarios for Java serialization in storm:
(1) After the client submits the topology task, Storm will serialize the topology task and send it to Nimbus;
(2) Supervisor after obtaining the task information from zookeeper, download the serialized file and jar package from Nimbus, start the worker process and deserialize the topology object when the task is submitted.

5. Summary
These are just a brief introduction to the Zookeeper cluster, Thrift Service Framework, ZEROMQ message Queue, Java serialization technology and its application in storm, followed by a deeper analysis of Storm's source code. If the above content, if not, welcome you to correct me.

Summary of Storm distributed real-time flow computing framework related technologies