Acker workflow for source code analysis of Twitter Storm

Overview

We know that a very important feature of storm is that it can ensure that every message you send will be completely processed. Complete processing means:

A tuple is fully processed, which means that this tuple and all tuple caused by this tuple are successfully processed. However, a tuple is considered to have failed to be processed if the message fails to be processed within the time specified by timeout.

That is to say, we will be notified of the success or failure of any spout-tuple and all its descendants. If you do this, you can see how Twitter Storm ensures that messages are not lost. From that article, we can know that storm has a dedicated acker to track the completion of all tuple. This article will discuss the detailed working process of acker.

Source code list

The source code involved in this article mainly includes:

1. Backtype. storm. daemon. acker
2. Backtype. storm. daemon. task
3. Backtype. storm. task. OutputCollectorImpl

Algorithm Overview

Acker's tracking algorithm for tuple is one of the major breakthroughs of storm. This algorithm enables any large tuple tree to be tracked at a constant length of 20 bytes. The principle is simple: acker saves an ack-val check value for each spout-tuple. Its initial value is 0, and each time a tuple/ack is sent, a tuple is sent, the tuple id must be different from the checksum value and be updated to the new ack-val value. Assume that every transmitted tuple is ack, And the last ack-val must be 0 (because a number is different from itself or the obtained value is 0 ).

Enter the subject

Next we will look at the source code to see which components will send messages to acker to complete this algorithm together. The following code processes messages by acker:

 

0102030405060708091011

(let [id (.getValue tuple 0)^TimeCacheMap pending @pendingcurr (.get pending id)curr (condp = (.getSourceStreamId tuple)ACKER-INIT-STREAM-ID (-> curr(update-ack id)(assoc :spout-task (.getValue tuple 1)))ACKER-ACK-STREAM-ID (update-ackcurr (.getValue tuple 1))ACKER-FAIL-STREAM-ID (assoc curr :failed true))]...)

Spout sends messages to acker when creating a new tuple.

Message formattuple.getValue())

 

1	(spout-tuple-id, task-id)

The streamId of the message is__ack_init(ACKER-INIT-STREAM-ID)

This tells acker that a new spout-tuple has come out. You can trace it, it is created by a task whose id is task-id (this task-id will be used later to notify this task that your tuple has been processed successfully/failed ). After the message is processed, acker will add such a record in its pending map (type: TimeCacheMap:

 

1	{spout-tuple-id {:spout-task task-id :val ack-val)}

This is the core data structure for acker to track spout-tuple. For each trace of the tuple tree generated by spout-tuple, you only need to save the above record. After acker, it will check when val is changed to 0 and 0, which means that all tuple generated by this spout-tuple is processed.

Will bolts send messages to acker when launching a new tuple?

When a bolt launches a new tuple, it will not directly notify acker. If so, three messages will be sent for each message:

1. When Bolt creates this tuple, it sends it to the next bolt message.
2. The message sent to acker when Bolt creates this tuple.
3. Ack message sent during ack tuple

In fact, storm only contains the first and third messages. It saves the second message. How can this problem be solved? Storm is a clever practice. Bolts saves the relationship between the new tuple and its parent tuple when launching a new bolt. Storm then sends the id of each tuple to be ack and the variance or value of all newly created tuple IDs to acker when each tuple is ack. This saves a message for each tuple (For details, refer to the next section ).

Send a message to the acker when the Tuple is ack.

Each tuple sends a message to the acker when it is ack. The message format is:

 

1	(spout-tuple-id, tmp-ack-val)

The streamId of the message is__ack_ack(ACKER-ACK-STREAM-ID)

Note that the tmp-ack-val here is the result that the id of the tuple to be ack is different from the id of all newly created tuple:

 

1	tuple-id ^ (child-tuple-id1 ^ child-tuple-id2 ... )

We can see this from the send-ack method in task. clj:

 

01020304050607080910111213

(defn- send-ack [^TopologyContext topology-context^Tuple input-tuple^List generated-ids send-fn](let [ack-val (bit-xor-vals generated-ids)](doseq [[anchor id] (.. input-tuplegetMessageIdgetAnchorsToIds)](send-fn (Tuple. topology-context[anchor (bit-xor ack-val id)](.getThisTaskId topology-context)ACKER-ACK-STREAM-ID)))))

Heregenerated-idsThe parameter is the id of all the child tuple of the input-tuple. From the code, we can see that storm will send an ack message to each spout-tuple of the tuple.

Why?generated-idsIs the sub-tuple of input-tuple? This send-ack is called by the ack method in OutputCollectorImpl:

 

1234567

public void ack(Tuple input) {List generated = getExistingOutput(input);// don't just do this directly in case// there was no output_pendingAcks.remove(input);_collector.ack(input, generated);}

Generated is composedgetExistingOutput(input)After calculation, let's take a look at the definition of this method:

 

123456789

private List getExistingOutput(Tuple anchor) {if(_pendingAcks.containsKey(anchor)) {return _pendingAcks.get(anchor);} else {List ret = new ArrayList();_pendingAcks.put(anchor, ret);return ret;}}

_pendingAcksWhat is stored in it?

 

010203040506070809101112131415161718192021222324252627282930

private Tuple anchorTuple(Collection< Tuple > anchors,String streamId,List< Object > tuple) {// The simple algorithm in this function is the key// to Storm. It is what enables Storm to guarantee// message processing.// What this map stores is the Sping from spout-tuple-id to ack-val.Map< Long, Long > anchorsToIds= new HashMap<Long, Long>();// Anchors is actually all of its fathers: spout-tupleif(anchors!=null) {for(Tuple anchor: anchors) {long newId = MessageId.generateId();// Tell every father that you have another son.getExistingOutput(anchor).add(newId);for(long root: anchor.getMessageId().getAnchorsToIds().keySet()) {Long curr = anchorsToIds.get(root);if(curr == null) curr = 0L; // Update the ack-val of spout-tuple-idanchorsToIds.put(root, curr ^ newId);}}}return new Tuple(_context, tuple,_context.getThisTaskId(),streamId,MessageId.makeId(anchorsToIds));}

We can see from the red section in the code above,_pendingAcksWhat is maintained in it is the correspondence between tuple and his son.

When Tuple fails to process, it will send a failure message to the acker.

Acker ignores the message content of the message (the streamId of the message isACKER-FAIL-STREAM-ID), Directly mark the corresponding spout-tuple as failed (the top 9th lines of code)

Finally, Acker sends a message to notify the Worker corresponding to spout-tuple.

Finally, acker notifies the task corresponding to spout-tuple Based on the processing results of the preceding messages:

010203040506070809101112131415161718192021222324

(when (and curr(:spout-task curr))(cond (= 0 (:val curr)); Ack-val = 0 indicates that all descendants of this tuple are; The processing is successful (all ack messages are sent); Then, the spout-tuple task is created when the message is successfully sent.(do(.remove pending id)(acker-emit-direct @output-collector(:spout-task curr)ACKER-ACK-STREAM-ID[id])); If the spout-tuple fails to be processed; Send an error message to the task that creates the spout-tuple.(:failed curr)(do(.remove pending id)(acker-emit-direct @output-collector(:spout-task curr)ACKER-FAIL-STREAM-ID[id]))))

Recommended reading:

Twitter Storm installation configuration (cluster) Notes

Install a Twitter Storm Cluster

Notes on installing and configuring Twitter Storm (standalone version)

Storm practice and Example 1