Kafka Source Code Analysis (ii) Metadata data structure and reading and updating strategies

First, the basic idea

The basic idea of asynchronous send is: When send, Kafkaproducer put the message to the local message queue recordaccumulator, then a background thread sender keeps looping and sends the message to the Kafka cluster.

To achieve this, there must be a precondition: that is, kafkaproducer/sender need to get the configuration information of the cluster metadata. The so-called metadata, that is, in the previous article, topic/partion and broker mapping relationship: Every Topic each partion, to know its corresponding broker list is what, leader is who, follower who.

Two, 2 data streams

So in, there are 2 data streams:

Metadata Stream (A1,A2,A3): Sender obtains information from the cluster and then updates metadata; Kafkaproducer reads metadata first and then puts the message in the queue.

Message Flow (B1, B2, B3)

As can be seen, metadata is read by multiple producer threads, a sender thread is updated, so it must be thread safe

III. Thread Safety of metadata

As you can see from the code below, all of its public methods are synchronized:

1  PublicFinalclassMetadata {2 ..... 3      Publicsynchronized Cluster Fetch () {4         return  This. cluster;5     }6      PublicSynchronizedLongTimetonextupdate (Longnowms) {7 ..... 8     }9      PublicSynchronizedintrequestupdate () {Ten .....  One     } A .....  -｝

IV. Data structure of metadata

1  PublicFinalclassMetadata {2 ...3     PrivateFinalLongREFRESHBACKOFFMS;//In the case of an update failure, the compensation time for the next 1 updates (this variable is not too significant in the code)4     PrivateFinalLongMetadataexpirems;//key values: How often, updates. The default is 600*1000, which is 10 minutes.5     Private intVersion//each update succeeds 1 times, and version increments by 1. This variable is mainly used in the while loop, wait time, as a cyclic judging condition6     Private LongLastrefreshms;//Last Update time (also contains the case of an update failure)7     Private LongLastsuccessfulrefreshms;//The time of the last successful update (2 equals if each succeeds). Otherwise, lastsuccessulrefreshms < LASTREFRESHMS)8     PrivateCluster Cluster;//Cluster configuration information9     PrivateBoolean needupdate;//whether to force refreshTen , One   ... A}

Kafka Source Code Analysis (ii) Metadata data structure and reading and updating strategies