Storm system architecture and code structure learning

Transferred from:http://blog.csdn.net/androidlushangderen/article/details/45955833

Storm Learning Series: http://blog.csdn.net/Androidlushangderen/article/category/2647213

Objective

What is storm,storm to do, a simple description that you can understand as a "quasi-real-time" Hadoop,hadoop is dedicated to offline data processing, while Storm makes up for Hadoop's shortcomings in this area, he is a real-time data processing platform. As for the various origins of storm, it is not a nonsense here, before I wrote an article about part of the storm's code structure, details click here .

Storm's overall architecture

First, to understand the overall structure of storm, a few key words in Storm Topology,spout,bolt,worker,executor,task, for the moment. These keywords represent the object is running in 2 kinds of nodes, a called Nimbus node, there is a node called supervisor, Storm is also the master/slave of the architecture, so there will be a single point of the problem. Both the Nimbus node and the supervisor can quickly fail and recover, and they are stateless, taking advantage of the metadata stored in the zookeeper. In each supervisor node, you can start a lot of workers, and in each worker, you can start a lot of executor actuators, each executor will be divided into a lot of task,task is the smallest unit allowed by the system. Here is one of them.

The overall structure of storm diagram is as follows, spout is the data source, like the faucet is generally constantly generating data, sent to the bolt node to execute.

Storm's meta-data storage

Storm uses zookeeper to store metadata between nimbus,supervisor and the internal workers, so you can perform an exception recovery. How is stored in the zookeeper, the answer is the existence of Znode, Znode is a tree structure, he has a root node, leaf nodes and so on, and the storm of these data is present in the leaf nodes, and finally according to the path to make the distinction and data read. It is equivalent to looking down from the root path, similar to the construction of the trie tree. Here is a schematic.

The code structure of storm

Storm's source code composition is mainly done by Clojure and Java2 languages. Clojure in the Baidu Encyclopedia on the introduction is that he is a running on the Java platform of a dialect, and the language of the grammar rules is very simple, after looking at the storm source code in the Clojure part of the codes, you should see a lot of functional programming traces. Clojure code implementation of the main functions include nimbus,supervisor,worker and other job execution related operations, of course, involves zookeeper related data exchange operations. The Java part of the code is the implementation of a lot of storm advanced properties, such as Trident part of the code, Trident is the storm of real-time message processing of the higher level of abstraction, I have not yet used. In the case of general job execution, the core code used is actually written in Clojure.

Storm system architecture and code structure learning