Storm series (13) Architecture Analysis worker-Maintenance ZMQ Connection

The worker calculates that the emitted message is received by those tasks according to the definition of topology and assigned to its own task, and because the tasks assigned on the worker may be adjusted, the worker needs to update the connection information periodically.

ZMQ Connection Information Update

Function: Mk-refresh-connections

Method Prototypes:

1	(defn mk-refresh-connection [worker ])

Method Description:

2. Call Worker-outbound-tasks to return the TaskID collection (outbound-tasks) that receives data from the worker parameter.
4. Define the This function and the corresponding callback function, and register yourself in the timer: Refresh-connections-timer.
6. The callback function with the callback parameter is used to listen for zookeeper corresponding node changes and synchronize.
8. Call Storm-cluster-state's assignment function to get the task assignment for the topology that corresponds to Storm-id.
10. Call: Executor->node+port stores executor to node+port mappings, and then calls To-task->node+port executor TaskID to TaskID based on node+ collection The mapping of the port, using the Select-keys function and the Outbound-tasks collection, to get the TaskID to Node+port from which the worker received the message, and finally to call Map-val and endpoint-> The string function gets the node+port of the strings, and the end result is a taskid to node+port hash table (my-assignment).
12. My-assingmnet is filtered to remove all taskid results that exist for the worker to Needed-assignment.
14. Gets the target worker collection (Node_port) from the Needed-assignment store to Needed-connections, Gets the TaskID collection on the target node from the Needed-assignment store to Needed-tasks.
16. Gets a list of all the keys in the worker node that are cached in the hash table from Node+port to ZMQ sockets, stored in current-connections.
18. Determine which connections need to be created and which can be closed, and save them to New-connections and remove-connections respectively.
20. Call the Msg/connect method to create a new connection based on Node+poer in new-connections and update to Cached-node+port.
22. Update the Cached-task->node+port to My-assignment.
24. Call the Close method of the socket you want to delete and remove the sockets from the: Cached-node+port.

Application process

1	: Refresh - Connections - Timer (Mk-halting-timer)
2	Refresh - Connections (Mk-refresh-connections worker)
3	_ (Refresh-connections nil)
4	(Schedule-Recurring (:Refresh-Connections-Timer worker) 0 (Conf TASK-Regresh-POLL-SECS) Refresh-connections).

Code Description:

1. Call Mk-halting-timer to create a timer that is called when the worker data is created.
2. Create a function to update the connection, and then immediately execute the refresh-connections function update ZMQ, and then execute the function continuously, task-refresh-secs the default is 10 seconds, and is called in the Mk-worker function.

Get topology active from zookeeper

The refresh-storm-active function gets the state information of the topology.

Method Prototypes:

1	(defn Refresh-Storm-active ([worker] ) ...)

Method Description:

1. Provides an anonymous function as the callback parameter, and callbacks when the corresponding zookeeper node changes.
2. By: The Refresh-active-timer timer completes the periodic call, which defaults to 10 seconds.
   (Shcedule-recurring (: Refresh-active-timer worker) 0 (conf task-fefresh-poll_secs) (partial refresh-active worker).
3. Call: Storm-cluster-state's Storm-base method gets the underlying information for topology.
4. Determine if the topology is active (: Active) and store the result in: Storm-active-atom variable.

receive function in worker

The MK-TRANSFER-LOCAL-FN function is used to generate and send messages to executor's receive queue, which is passed between executor within the same worker through the function.

Method Prototypes:

1	(defn mk-transfer-local-fn [worker])

Method Description:

1. Call: Short-executor-receive-queue-map Returns the mapping of the taskid of the first task in executor to the corresponding receive queue for that executor. Save to the Short-executor-queue-map variable.
2. Call Task->short-executor to return the mapping from TaskID in the worker to the taskid of the first task in executor.
3. Defines the returned anonymous function, which enters a set of message Tuple-batch received for ZMQ, grouping the message according to the taskid of the first task in the executor corresponding to the message taskid. The variable grouped corresponding key is the taskid of the first task in executor, and the value is a group of messages belonging to that executor.
4. Receive Message queue Q that corresponds to TaskID and executor is obtained by Short-executor-receive-queue-map.
5. Call the Disruptor/publish method to send the received message to the queue Q.

The Send function in Woker

MK-TRANSFER-FN is used to send data for executor in the following two scenarios:

1. The target TaskID and the sending TaskID belong to the same worker, which does not need to transmit messages across processes, sending messages through MK-TANSFER-LOCAL-FN to the receive queue executor the receiving side.
2. The target of the message is taskid with the sending TaskID in a different worker, and the message is serialized (Kryotupleserializer) and sent to the worker's send queue. The worker (Mk-tranfer-tuples-handler) is responsible for sending messages from the queue through ZMQ.

Method Prototypes:

1	defn mk- transfer [worker]

The method returns a function that is a parameter to the serialization serializer and a set of messages.

Communication between different worker's

There is an additional thread in the worker that listens to the Transfer-queue (worker's message send queue) and Mk-transfer-tuples-handler is used to create the corresponding message processor.

Method Prototypes:

1	defn   MK - transfer - tuples - Handler   [worker]

1. Call Cached-node+port to get the ZMQ socket connection in the worker that corresponds to the target node+port, and save to Node+port->socket.
2. Call Worker-data's cached-task->node+port to get TaskID to node+port mappings and save to Task->node+port.
3. Define a Clojure-handler, the corresponding function is defined as FN [packets _ Batch-end?], the first parameter is a set of messages packets, the second one is ignored, and the third is the end tag.
4. Call the Msg/seng function to send the message out.

To send the startup of a listener thread:

1	Transfer - tuples  (Mk-transfer-tuples-handler worker)
2	Transfer - Thread (disruptor /consume-loop * (:transfer-queue worker) Transfer-tuples)

Storm series (13) Architecture Analysis worker-Maintenance ZMQ Connection