Kafka shutdown stop shutdown very slow problem solution

Kafka shutdown stop very slow problem

When the amount of data is large, consumer data fetching data a lot, there may be a lot of data into the business processing,

Suppose that once poll has 10,000 data to enter the business process, and the business program is bound together with the poll thread synchronous execution, assuming the average per piece of data, the execution of the business process costs 100ms,

Then poll the data once, at least to perform 1w*0.1s = 1000s = 16.67 minutes.

So, when the amount of data is large, stopping a thread (which needs to wait for the business program to finish processing the data) can take up to more than 10 minutes.

Shutdown Problem Solving Solution

1, to the asynchronous processing of data, consumer taken out of the data, put into the Blockqueue, by the asynchronous thread to handle, when the asynchronous thread does not work, block consumer, call Consumer.pause () method Avoid group Management rebalance, code as follows (from Spring-kafka):

1 2 3 4 5 6

Avoid group management rebalance due to a slow consumer this. Consumer.pause (the. Assignedpartitions.toarray (New to   picpartition[this. assignedpartitions.size ())); public void onpartitionsassigned (collection<topicpartition> partitions) {this. assignedpartitions = Partitio ns }

2, if it is synchronous execution of data processing, consider increasing the speed of the business process of processing.

3, synchronous processing of data, but changed to manually submit offset, when shutdown, poll data does not need to process all, only need to record the location of processing. The code examples are as follows:

1 2 3 4 5 6 7 8 9

List data = consumer.poll (); for (record :  data)  {       if (shutdown)  {  //  stop immediately after receiving the shutdown command, Unhandled data will be discarded          break;