Kafka Consumer Code Research and Core Logic analysis

Kafka Consumer API is the interface of the client, encapsulates the receipt of messages, heartbeat detection, Consumer rebalance, etc., the code of this analysis is based on the kafka-clients-0.10.0.1 Java version

Kafkaconsumer.pollonce is the polling entry that completes a polling action, including all the logic related to consumer, with the following logical process:

Further, the related process is expanded as shown in:

The red wireframe indicates the active process of pollonce in one poll, and the right side is the corresponding expanded activity process, in pollonce is the key method of consumer, all related logic is implemented in this method, including message pull, heartbeat detection, consumer rebalance, Automatic submission of offsets and update operations, and so on.

1: Get Coordinator:Ensurecoordinatorready, request for Groupcoordinatorrequest

For the group where the consumer is located (different group is differentiated by GroupID), a coordinator is found from all the brokers, and the user is initially configured with a default broker list, which finds a node with the least load recently. After the request Groupcoordinatorrequest is constructed, it is placed in the Consumernetworkclient unsent queue and then blocks the poll (future) method that calls Consumernetworkclient. Until the future IsDone

PrivateRequestfuture<void> sendgroupcoordinatorrequest() {Node node = This. Client. Leastloadednode (); //Find a minimum load node     ...... Groupcoordinatorrequest metadatarequest=New groupcoordinatorrequest( This. groupId); returnclient.send (node, apikeys.group_coordinator, Metadatarequest). Compose (NewRequestfutureadapter<clientresponse, void>() {@Override Public voidOnsuccess (clientresponse response, requestfuture<void>Future )         { Handlegroupmetadataresponse(response, future);   }     }); }}Private void Handlegroupmetadataresponse(Clientresponse resp, requestfuture<void>Future ) {   ...... Client.tryconnect (Coordinator); //Connection Coordinator //start sending heartbeats only if we have a valid generation   if(Generation > 0) Heartbeattask.reset (); //If generation >0 indicates that the heartbeat delay task is set after coordinator is reconnected Future.complete (NULL);   ......}

Before Kafka 0.9, the consumer group was maintained by ZK, but because of the "herd" and "split brain" problems, after redesign, in the new version by the broker cluster to select a node as coordinator, To resolve synchronization of individual consumer in group, such as Rebalance,failover,partition Assignment,offset Commit

Refer to Kafka consumer design to reconstruct the original text:

Https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design

Https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal

2: join group, assign partition and synchronize group status and load balancer :ensurepartitionassignment, request is joingrouprequest and Syncgrouprequest

After getting coordinator, call Ensurepartitionassignment, and continue to call the Ensureactivegroup method internally, the main function of this method is join group and Sync Group. If there are still non-issued requests (unsent and inflight queues) on the coordinator node before preparing to send the Joingroup request to coordinator, then all requests, such as blocking, need to be blocked before proceeding. Sendjoingrouprequest constructs a good joingrouprequest and puts it into the unsent queue, where the callback class is passed in to handle the response

 PrivateRequestfuture<bytebuffer> sendjoingrouprequest() {if(Coordinatorunknown ())returnrequestfuture.coordinatornotavailable (); //send a join group request to the CoordinatorLog.info ("(re-) joining group {}", groupId); Joingrouprequest Request=New joingrouprequest(groupId, This. Sessiontimeoutms, This. MemberID, ProtocolType (), metadata ()); Log.debug ("Sending Joingroup ({}) to coordinator {}", request, This. Coordinator); returnclient.send (Coordinator, Apikeys.join_group, request). Compose (New Joingroupresponsehandler()); }

Block the call in the later Client.poll (future) until coordinator returns the result, Callback handler function Joingroupresponsehandler.handle, if the return result error code is Errors.none, it indicates that successfully joined the group, if the return result indicates that consumer is leader, you need to continue in Onjoinleader, by L Eader allocates partition information and tells coordinator to synchronize to other follow. In the case of follow, the synchronization message is sent in the Onjoinfollower

Private classJoingroupresponsehandlerextendsCoordinatorresponsehandler<joingroupresponse, bytebuffer> {@Override Public voidHandle (Joingroupresponse Joinresponse, requestfuture<bytebuffer>Future ) {       ......if(Error = =errors.none) {           ......           if(Joinresponse.isleader ()) { Onjoinleader(joinresponse). Chain (future); } Else{ onjoinfollower(). chain (future); }         } Else if(Error = =errors.group_load_in_progress) {         ......
......    } }  PrivateRequestfuture<bytebuffer> onjoinfollower() {Syncgrouprequest request =Newsyncgrouprequest (groupId, ...); //sync Group request
returnsendsyncgrouprequest (Request); }  PrivateRequestfuture<bytebuffer> onjoinleader(joingroupresponse joinresponse) {Try {        //perform the leader synchronization and send back the assignment for the groupmap<string, bytebuffer> groupassignment = performassignment(...);        //leader Assigning Partitions Syncgrouprequest Request=New syncgrouprequest(...); //leader Sync Group return sendsyncgrouprequest(Request); } Catch(RuntimeException e) {returnRequestfuture.failure (e); }  }

In Onjoinleader, call the Performassignment method to assign the group based on the Group protocol (such as Range,roundrobin) configured by the broker Member consumes the Topicpartition, then sends the synchronization request Syncgrouprequest to coordinator, while the other group member is follow, also sends the request, Get the corresponding allocation status from coordinator, after completing Joingroup and syncgroup, partition distribution status in Onjoincomplete update

3: Update pull offset: updatefetchpositions

See https://www.cnblogs.com/benfly/p/9830784.html Offset Management

4: Perform deferred task: Executedelayedtasks

Deferred tasks include Autocommittask and heartbeattask, deferred tasks are tasks that are performed every other cycle, the duration of the auto-commit task is auto.commit.interval.ms, and the period of the heartbeat task is heartbeat.interval.ms, deferred tasks are saved in Delayedtaskqueue in the delay queue, and deferred tasks, such as commit offsets or heartbeat detection, are performed after the specified period has been reached.

The auto-submit task and heartbeat task implement the delay task interface and implement the task running method run

A task in the deferred queue, which invokes the Run method in each poll, performs a specific task

• Automatically submit tasks

When Kafkaconsumer is instantiated, a consumer coordinator object is created,

Private kafkaconsumer (consumerconfig config,                          deserializer<K> keydeserializer,                          deserializer <V> valuedeserializer) {        try  {            thisnew Consumercoordinator (This//create consumer Coordinator ...          }}

In the consumer coordinator Consumercoordinator, there is an auto-submit task member

 Public Final class extends abstractcoordinator {    privatefinal//Auto-Submit Task Object }

During the creation of the consumer coordinator object, if the default configuration is auto-commit, the autocommit task is initialized and a commit task is set

 public   Consumercoordinator ( Consumernetworkclient Client, ...) {       .......         if   (autocommitenabled) {//if the task is configured to autocommit, the Autocommit task object is initialized   this . Autocommittask = new             Span style= "COLOR: #000000" > Autocommittask (Autocommitintervalms);  this  .autocommittask.reschedule (); //add a task in the delay queue, set the delay execution time }  else   { this . Autocommittask = null  ; }        ......    }

 Public class Implements closeable {    privatefinalnew//delay queue, save auto-commit task item and Heartbeat Task item     publicvoidlong  in) {        Delayedtasks.add (task, at);}    }

• Heartbeat Detection Task

After a successful consumer joingroup, the heartbeat task is set to begin

 Public voidEnsureactivegroup () {... while(Needrejoin ()) {Ensurecoordinatorready ();            ...... Requestfuture<ByteBuffer> future =sendjoingrouprequest (); //apply to join Group Future.addlistener (NewRequestfuturelistener<bytebuffer>() {@Override Public void onsuccess (bytebuffer value) {//handle Join Completion in the callback so, the callback would be invoked//even if the consumer is woken up before finishing the rebalanceOnjoincomplete (Generation, MemberID, protocol, value); Needsjoinprepare=true; Heartbeattask.reset ();                //Join group success, set heartbeat task } @Override Public voidonfailure (runtimeexception e) {//We handle failures below after the request finishes. If the join completes//After have been woken up, the exception is ignored and we'll rejoin                }            });    ......        } }

5: Message fetching and consumption

See https://www.cnblogs.com/benfly/p/9830784.html messages for pull and consumption

Kafka Consumer Code Research and Core Logic analysis