Adaptive dynamic Flow control algorithm for storm

• Objective

Business of the log ETL frame has been a lot of problems, each time a failure caused by hand busy chaos, so this determination to make a big makeover. This ETL system is based on storm implementation, mainly rely on spout pull the original log, bolt processing and then storage, in order to improve the throughput, the use of 12 bolts for parallel processing. The old algorithm is not using the ACK feature of storm, and it is sent to the corresponding bolt according to the hash value of sending tuple, the load balance problem is not considered at all, and the log of the day will need to be re-processed after the crash reboot, and the usability is very low.

• New ideas.

In order to be able to achieve the load balance of bolt, think of using dynamic equilibrium algorithm to adapt, also need to provide a reliable checkpoint mechanism, can make the crash restart from the current point in time to continue processing.

This dynamic balancing algorithm can refer to the TCP flow control algorithm. TCP is a reliable message transmission protocol with reliability, high efficiency and the ability to avoid network congestion and other characteristics. Here are the features of several major TCP flow control algorithms:

tcp combine multiple bytes of sending information into a single datagram for increased delivery efficiency
1. TCP uses a time-out resend mechanism and is sent according to the priority of the timeout period.
2. The receiver receives the datagram and sends an ACK to the corresponding send side, here is the cumulative ACK algorithm, that is, if the datagram 2 is received sequentially, 3, then send two Ack,ack is the datagram 1.
3. Send side has a send window, the receiver also has the concept of receiving window, the sending window indicates the number of datagrams that can be sent by the sender, The receive window can be understood as the receiving side of the cache queue for receiving datagrams, typically two windows in the same size.
4. The receiver can send the size of the receiving window to the sending side according to its processing speed, the sending side adjusts the sending window size to avoid the sending speed exceeding the processing speed.

Of course, TCP is far more than the 6 points of the characteristics, here is only part of the new algorithm has a reference to the characteristics. Refer to here and the IETF standard specification rfc2001.

• Specific algorithms

Because the ETL system relies on storm to develop, so there are certain restrictions. However, the TCP algorithm can be analogous to storm's characteristics. Spout, which is the sending side, each sending unit is called a tuple and can be likened to a TCP datagram. When a tuple is received by the bolt receiver, an ACK or fail-off of the tuple is required, and then spout will resend the fail tuple, and in the same way storm will detect a tuple that is not responding and send fail to spout.

Although Storm has these delivery features, Storm has no way to send more feedback to spout, in addition to the ACK and fail, in particular the size of the Bolt's receive window (that is, the remaining capacity of the cache queue), so that spout cannot send a window adjustment based on the bolt. In addition, each bolt can only ACK or fail corresponding to the tuple, can not use the cumulative ACK algorithm. In addition, the relationship between spout and Bolt, is currently a one-to-many, that is, a sender sent to multiple receivers for parallel processing, so load balancing is particularly important.

      For this new algorithm, the goal is to achieve high throughput, high availability, and simplicity. High throughput is the amount of concurrent processing of multiple bolts, and in order to improve throughput, the key is to achieve dynamic load balancing. For each bolt, the larger the sending window is not necessarily the better, because even if sent in the past, Bolt may not be able to deal with it immediately, instead, should be sent more tuples to deal with faster bolts up, so the most appropriate approach is based on the speed of the bolt processing, That is, the ACK round trip time to measure the Send window size. For high availability, a mechanism similar to checkpoint is required for the processing points to be re-processed after a reboot to be able to crash before the joiner crashes. To achieve this requirement, and to ensure simplicity, you can save an already-Ack RT value. However, because the bolt can only do ack or fail to receive the tuple, it must be in the spout-end statistics using the cumulative algorithm of the RT value, that is, if the ACK of the 1,3,4 tuple, can only record the 1th tuple of the RT value, if you receive an ACK 2 of a tuple To record the RT value of the 4th tuple. Simplicity is to avoid the coupling of the algorithm and the code of the actual business, which can be used similar to the consumption-producer model.

In the RFC2001 specification, a heuristic dynamic flow control algorithm is adopted for TCP congestion processing and avoidance algorithm. As the network congestion can not be used to accurately determine the digital, but can be simply based on the receiver's ACK and packet loss to determine the congestion situation, and then linearly or exponentially increase/decrease the size of the sending window, so as to avoid increased network congestion pressure. Based on this algorithm, a similar adaptive dynamic Flow control algorithm can be extended according to the target of joiner:

• spout combines multiple database records into a single tuple to send, which reduces the pressure at ack/fail and improves delivery efficiency

• spout have different send windows for different bolts, each sending window will dynamically increase or decrease according to the situation, and will record the sending time of the tuple.

• Spout The ack of the bolt is received, the processing time value is calculated based on the sent time of the previous record, if it is in the normal range, the normal state, the Send window size is increased linearly, if the threshold is exceeded, the slow state is reduced, the Send window size is decreased linearly, In addition, spout calculates the RT value of a tuple that needs to be recorded by a cumulative ack.

• When spout receives a fail, that is, Storm determines the tuple processing time timeout (currently does not allow Bolt to actively send fail), then into the busy state, immediately reduce the size of 1 send window, and put the tuple back to the send buffer, waiting to send to the next bolt, Reduce the pressure on the current busy bolts.

compared with TCP congestion control algorithm, this new algorithm reduces some features, such as Slow Start, Fast retransmit, some of which are not necessary (Slow start is to avoid congestion at first, but the initialization state of the bolt is definitely idle), and some are for the case of multiple receivers, (compared to Fast retransmit. Sent to the next free bolt is more advantageous), In addition, the new algorithm does not have Ssthresh, slow start threshold, to distinguish the exponential increase of the sending window and linearly increase the threshold of the sending window, because the exponential increase in the sending window when it is easy to create slow state, In addition, blindly increasing the size of the sending window does not improve throughput, so only the linear increase is selected. There is also bolt does not emphasize the processing order of the tuple, that is, as long as the ACK, the corresponding tuple from the sending window to move out, you can send a new tuple, this and the TCP Send window must wait to be processed to the right to move the difference.

• Realize

The entire implementation of the class diagram is as follows:
Flowproducer is a stream producer implemented by the business, mainly to realize the generation of data flow, Flowcontrolspout realize the spout logic of storm, need to send storm to the tuple and Ack/fail The message of a tuple is passed to Adapativeflowconsumer, which in addition acts as a buffer queue, and the primary buffer flowproducer the resulting data stream waiting for adapativeflowconsumer consumption. Adapativeflowconsumer is an adaptive dynamic flow control algorithm for consumers, each adapativeflowconsumer corresponding to a bolt, itself maintains the corresponding sending window size, as well as the re-hair mechanism. Flowcontrolbolt implements Storm's bolt logic, which is primarily responsible for the ACK of the processed tuple (processing failed send fail is currently not supported).

• Summarize

Compared with the old algorithm, the new algorithm lookswith higher throughput and availability. The specific amount of log processing has been improved. Still , the new algorithm has a certain level of elevation, which is more accurate load balancing control. At present, when the tuple is sent, the new algorithm is the round robin algorithm per request Adapativeflowconsumer send tuple, but in fact, should calculate the most idle bolt, according to the priority to send, This requires a measure of the degree of idle, but the design of the algorithm when considering the generation of the flow is uninterrupted, and the measurement of the idle algorithm for the time being simple, so it is temporarily not considered. Maybe you can add it when you need it in the future.

Adaptive dynamic Flow control algorithm for storm