Kafka shutdown stop shutdown very slow problem solution

Kafka shutdown stop very slow problem

When the amount of data is large, consumer data fetching data a lot, there may be a lot of data into the business processing,

Suppose that once poll has 10,000 data to enter the business process, and the business program is bound together with the poll thread synchronous execution, assuming the average per piece of data, the execution of the business process costs 100ms,

Then poll the data once, at least to perform 1w*0.1s = 1000s = 16.67 minutes.

So, when the amount of data is large, stopping a thread (which needs to wait for the business program to finish processing the data) can take up to more than 10 minutes.

 

shutdown Problem Solving solution

1, to the asynchronous processing of data, consumer taken out of the data, put into the Blockqueue, by the asynchronous thread to handle, when the asynchronous thread does not work, block consumer, call Consumer.pause () method Avoid group Management rebalance, code as follows (from Spring-kafka):

Avoid group management rebalance due to a slow consumerthis.consumer.pause (This.assignedPartitions.toArray (new Topicpartition[this.assignedpartitions.size ()]));p ublic void onpartitionsassigned (collection<topicpartition > Partitions) {    this.assignedpartitions = partitions;}

2, if it is synchronous execution of data processing, consider increasing the speed of the business process of processing.

3, synchronous processing of data, but changed to manually submit offset, when shutdown, poll data does not need to process all, only need to record the location of processing. The code examples are as follows:

List data = Consumer.poll (); for (record:data) {     if (shutdown) {  ////shutdown command stops immediately after receipt, unhandled data will drop break        ;    }   deal (record);   Savetopicoffset (record);} Submitdealtdataoffset ();

Other than that

Kafka can't stop, shutdown can't shut down.

The reason is that it is stuck in the Consumer.close () method, it will submit the offset information, if the network is interrupted or the Kafka server has a problem that can not be submitted offset, The Consumer.close method will always be stuck (the loop attempts to commit offset and never interrupt).

See also: Kafka poll has been waiting for a bug:

https://issues.apache.org/jira/browse/KAFKA-4189?jql=project%20%3D%20KAFKA%20AND%20resolution%20%3D% 20unresolved%20and%20component%20%3d%20consumer%20order%20by%20priority%20desc

https://issues.apache.org/jira/browse/KAFKA-3172?jql=project%20%3D%20KAFKA%20AND%20resolution%20%3D% 20unresolved%20and%20component%20%3d%20consumer%20order%20by%20priority%20desc

Workaround: There is no good way, can only change the automatic submission of offset to manually submit offset. However, I wrote a program that can forcibly kill a thread after calling Consumer.close as a temporary solution.

Kafka shutdown stop shutdown very slow problem solution