Application of XOR Algorithm in storm Reliability

1. first look at the difference or in Mathematics

XOR or XOR is a mathematical operator. It is applied to logical operations. The XOR symbol is "^ ".

An exclusive or semi-addition operationAlgorithmIt is equivalent to a binary addition without bits. In binary, 1 indicates true, and 0 indicates false. Then, the XOR algorithm is as follows: 0 or 0 = 0, 1, or 0 = 1, 0, or 1 = 1, 1 = 0 (0, 1 ),

Since the result of the same object XOR operation is 0, there is such a formula,

A xor B... Xor B XOR a = 0, where each operand appears only twice.

2. Storm reliability mechanism

In storm, a system-level component is Acker. Acker traces whether the tuple generated by the stream ID (msgid) from spout is complete in each task. After spout or bolt finishes processing tuple, it will tell Acker that I have processed the source tuple (such as tupleid = 1). If emit is a tuple, at the same time, it will tell Acker that I have fired a tuple (for example, tupleid = 2). If a large number of highly concurrent messages occur, the traditional method of tracking execution in the memory is as follows, memory overhead can be very large, or even memory overflow. Acker cleverly utilizes the XOR mechanism and only needs to maintain a mark bit of msgid. The solution is that during the initial process of Acker, initialize an ack-Val (0) Check value for each msgid.
During tuple, the system performs XOR operations on these two values first, and then generates XOR with the current check value ack-Val in Acker to generate a new ack-Val value, when all tuple processing is completed and confirmed, the final ack-Val is naturally 0 (because every tuple, from emit to ACK, is processed twice in XOR operations, so the final result is 0, which can be verified by the formula above ).

See: