Storm series (16) architecture Analysis Executor-bolt

Prepare the data for the message loop

Function Prototypes:

1	let[executor-sampler (Mk-stats- Sampler (:Storm- conf Executor-data)]

Main functions:

Defines the TUPLE-ACTION-FN function, which obtains the corresponding Bolt object based on TaskID and calls its executor method.

Bolt Input handler function

Function Prototypes:

1	tuple - Action - fn (fn [Task-ID ^tupleimpl tuple] )

Main functions:

Get the bolt-obj for the bolt and call the executor callback method.

Message send function for Bolt

Function Prototypes:

1	Bolt - Emit  (fn [stream anchors values Task])

Main functions:

1. Call Tasks-fn to get the TaskID collection for the receiving end of the message.
2. Call the TRANSFER-FN function to send a message that is similar to the implementation in spout (the only difference is not to use the Overflow-buffer cache).

Initialization of Bolt objects

Call Bolt's Prepare function
Initialization process:

1. Gets the bolt object and defines the associated method. The Bolt-emit method is used to send messages to the executor message sending queue.
2. The prepare method of the bolt object is called, and the Outputcollector object of the Bolt object is instantiated as an incoming parameter of the prepare method, and the emit method of Outputcollector calls the Bolt-emit function to send the message. The ACK and Fail methods are used to trace the message.
3. Call the Mk-task-receiver function to get the handler function (TUPLE-ACTION-FN) for the receive queue.

Message loops

The consume-batch-when-available function that calls the blocking method processes the messages in the receive queue.

1	(FN [] (distruptor/comsume-Batch -when-available Receive- Queuu event-handler))

Create executor

Create the Executor function Mk-executor

Function Prototypes:

1	(Defn mk- executor [ worker executor- ID])

Method Description:

1. Call Mk-executor-data to create the executor data.
2. Call Mk-task to create the data corresponding to each task in the executor.
3. Call the Start-batch-transfer->worker-handler! method to start the executor data send thread.
4. Call the Mk-threads method to get the executor main loop thread and wrap the mk-threads through the With-error-reaction macro. When an exception occurs, call the Report-error-and-die method to log the error and exit.
5. Instantiates an Runningexecutor object to manipulate executor.
6. Instantiating shutdownable is used to exit executor and clean up related resources, including the following:
   A) end the Disruptorqueue message loop.
   b) End the startup thread in executor
   c) Clean up data for user hooks
   d) Disconnect the zookeeper from the connection
   e) Call the Close method of spout or bolt in executor.

Get the Grouping function

Function Prototypes:

1	(Defn Outbound - Components [^workertopologycontext Worker-context component- ID])

Function Description:

Gets the grouping function from the component to a stream, and the TASK-FN function obtains the target task collection for the message by calling the grouping function.

Function Description:

1. Call the Gettargets method of the Workertopologycontext object to get a hash table with the key of the current component corresponding to the stream, a hash table that records how the target component receives data from the stream.
2. Call the Outbound-groupings function to get the grouping function.

outbound-groupings
Definition of a function

1	(Defn- Outbound-groupings [^Workertopologycontext worker-Context  This-Component-ID Stream-ID  out- Fields Component->grouping])

Method Description:

1. Filter the target component, if the component corresponding to the TaskID set is empty, how to be filtered out.
2. The map function is used to process the component and its grouping, call the Mk-grouper function to generate the grouping function, and eventually return a hash table that holds the mapping relationship from component to grouping function.

Mk-grouper

Returns a function that returns a TaskID collection that represents the destination task set for the message to be sent

Function Prototypes:

1	(Defn Mk-Grouper [^Workertopologycontext Context Component-ID Stream-ID ^ Fields  out- Fields Thrift-Grouping ^List Target-tasks])

Function Description:

1. Gets the number and list of tasks that correspond to the target component, which are input to the function that computes the target task. Some groupings require only the number of tasks for the target component, such as: shufflegrouping operations.
2. For the thrift type, the grouping function is constructed separately for different grouping methods.

Trigger System Ticks

The setup-ticks! function periodically sends a tick message to executor's receiving message queue. Executor after the tick message is received, the Send queue's timeout operation is executed. Setup-ticks! is primarily used to operate on messages sent out by spout nodes.

Function Prototypes:

1	(defn setup-ticks [worker Executor -data])

Function Description:

1. The configuration item topology-tick-tuple-freq-secs is used to control how often the message is sent to the __system stream and __tick stream, and tick-time-secs is used to hold the frequency value. Receive-queue for executor corresponding receive disruptor Queue,context for Workertopologycontext objects, tick messages are sent only to local worker, Can not be received by the executor of the other worker.
2. Determines if the tick-time-secs is set and starts to set the system's tick message if it is set.
3. If the node is a spout node and the message timeout is not set, the print message exits. The parameter topology-enable-message-timeouts is used in debug mode, because a timed-out message can cause additional complexity to system debugging, so the timeout action of the message is temporarily closed during debugging. When spout receives a tick message, the data cached in the pending object can be timed out.
4. The timer is set for the interval using the user timer defined by the worker tick-time-secs, and a timer callback function is defined, and a message is sent to receive-queue that corresponds to a taskid of nil. Indicates that all the tasks in the executor receive the message, the content of the message is tick-time-secs,-1 represents the system TaskID, and the last item indicates that the message is sent to System-tick_stream.

Storm series (16) architecture Analysis Executor-bolt