Implementation of Kafka and EMS message batch ACK

We now use Kafka and EMS two ways to receive external messages, before receiving a message on the ACK, the system's current message volume is approximately billion per day, concentrated in the working hours of 8-10 hours. This means that each message ack consumes a lot of network resources and slows down the message processing speed. Therefore, the decision to use bulk ACK to reduce network consumption.

Implementation process encountered a problem, assuming that No 10 ack once, if there are 37 data, means that the first 10 can be successful ACK, and then 7 because there is not enough batchsize may have been not ack.

For Kafkaconsumer, the consumer invokes the poll method to get the message from the server actively, which accepts the timeout parameter. At this point, even if the batch size is not reached, there is still a chance to act when timeout.

And for EMS, there is a bit of trouble.

EMS client has two kinds of message receiving methods. One is that the client actively invokes the consume method to get the message from the server. This approach, like Kafka, can accept timeout parameters, so it's not a problem.

The problem is another way, onMessage () It is event-driven and does not accept timeout parameters. So that when the last 7 messages are received, it waits for the next message to arrive. At this point, the client has received and processed the first 7, but there is no chance to ACK, if the client is hung up, then the 7 processed messages have no chance to be ack. This causes the message to be repeatedly processed. The solution is to open a Guardian thread timed to do the ACK, because the EMS ack depends on the specific message, so each received a message to cache down, and a few last received the time, and then use this last message to do an ACK.

Kafka and EMS support the last message ACK, the preceding message automatically ack.