"Original" KAKFA cluster package source code Analysis

The Kafka.cluster package defines the basic logic concepts of Kafka: Broker, Cluster, partition, and replica--these are the most basic concepts. Only by understanding these concepts can you really use KAKFA to help fulfill your needs. Because the Scala file is not many, or the usual, we one analysis.

First, Broker.scalaBroker can be said to be the most basic concept of Kafka, without broker there is no Kafka cluster, not to mention the responsibility of decoupling producers and consumers. Kafka uses a case class to define the broker class. The attributes of a broker are ID, host, and Port, respectively, representing the broker's ID number, hostname or IP, and ports. In addition, several convenient methods are defined: 1. ConnectionString: Returns a string such as Host:port, if IPV6, returns the [host]:p ORT, In fact, in the specified producer core configuration item: Metadata.broker.list You also need to specify in this format, of course, multiple host:port pairs separated by commas 2. WriteTo: The property information of the broker is written to a bytebuffer, with the following layout: 4 bytes of ID information + 2 bytes of host name or IP length n information +n bytes of host name or ip+4 bytes of port information 3. Sizeinbytes:broker The total number of bytes in the underlying Bytebuffer, 2 (host name or IP length N information, 2 bytes) + N + 4 (Broker ID) + 4 (port). For example, a broker's message is ID = 1, host:port = localhost:9092, then the value of sizeInBytes is 2 + 9 + 4 + 4 = 19 bytes 4. The Equals:broker class has a copy of the Equals method, only id,host, port is equal, and host is equal to two broker 5. Hashcode: Similarly, to use both Id,host and port three properties to calculate hashcode In addition to this case class, the Scala file also defines an object that provides some factory methods for creating broker:1. Createbroker: Receives an ID and a JSON-formatted string to create the broker. The format of the string in JSON format is probably this: {"Jmx_port":-1, "timestamp": "1429495288971", " Host":" tc_207_97 "," Version ": 1," Port": 9092} then the method extracts the port and host from it. This string of strings is saved under Zookeeper node/brokers/ids/<borkerid> 2. Readfrom: A broker is also created, but this time the broker's properties are extracted from Bytebuffer (Bytebuffer written by the WriteTo method above) Second, Cluster.scalaThe so-called Kafka cluster is actually the currently available broker collection. Since more than one Broker,cluster class is involved, it is necessary to define a variable of the container class to hold the broker. Yes, it defines the brokers variable and uses a variable HASHMAP to do the underlying implementation. Key is the broker's id,value is the corresponding broker instance. There are two ways of creating a Kafka cluster: 1. Create an empty cluster instance, and then call the Add method to add broker;2 continuously. Passing a set of broker instances to cluster directly creates the Kafka cluster (secondary constructor). In addition, the cluster class defines several methods: 1. Getbroker: Find the corresponding Broker2 based on the ID number. Add: Add a broker to the cluster in 3. Remove: Removes a Broker4 from the cluster. Size: Returns the number of broker for the current cluster Third, Partition.scalaThe partition class is a data structure that holds a topic partition in the Kafka. Each partition has a leader partition and a set of ISR (In-sync replica) copies (note that the ISR also includes leader). All read and write operations are only passed to the leader partition, and the replicas in the ISR are referred to only as followers (follower). On this part, there is a good article on the Internet, recommend people to read: Http://www.infoq.com/cn/articles/kafka-analysis-part-2 This class has a lot of code, we follow the constructor, class member variables, The order of the class methods. The constructor is parsed first. 1. ConstructorsThe constructor for this class receives 4 parameters, which have the following meanings: I. Topic: Which topicii the partition belongs to. PartitionID: Partition ID Number III. Time: Provide service IV for the date. Replicamanager:replicamanager object (which will be said later), The Getorcreatepartition method in Replicamanager, which is used primarily for replica management, calls this constructor to create a new partition instance. 2. Class member Variables LocalbrokeridBroker ID number configured in the--config file, each broker ID number should be a unique positive integer Logmanager--Log Manager for creating/deleting logs, getting from Replicamanager zkclient--zookeeper Client for the operation of ZK, obtained from the Replicamanager Assignedreplicamap--The Map,key of a replica is that broker Id,value is a replica instance (represented by the replica class, which is said later in this class). Typically used to add local partitions to the corresponding map Leaderisrupdatelock--lock object, which provides synchronization protection for multithreaded reads zkversion--The state information of the partition on the zookeeper, the initial state is 0 Leaderepoch--the number of times the broker becomes partition leader, the initial value is-1 leaderreplicaidopt--The broker ID number of the leader copy of the partition, of course it may be none Insyncreplicas(ISR)--the set of ISR replicas for this partition Controllerepoch--controller Broker changes the number of times the partition leader, the initial value is 0 interpolation, the code appears in the epoch, the sense should be the epoch time, but the zookeeper is the integer value, the number of times logident--the number of times that the print log uses a string Newgauge ("underreplicated")--isr the number of collections is less than all replicas 3. Class Methods Getreplica--Gets the corresponding replica instance according to a replica ID, by default, gets the copy object corresponding to the local broker leaderreplicaiflocal--If the leader copy of the partition returns the corresponding copy on the local broker, none is returned. isunderreplicated--Compare the number of ISR and total copies isreplicalocal--Determine if a given copy is on a local broker addreplicaifnotexists--Add a given copy of the copy map Getorcreatereplica--Gets the replica instance if it does not exist directly to create a new replica. This is done based on the given replica ID in the copy map search, if there is a corresponding record directly return, otherwise, first determine whether the local copy, if not a local copy, directly create a replica object, added to the copy map returned. If it is a local copy, create the corresponding log and read the displacement information from the checkpoint file in the corresponding directory into a replica object, and add it to the copy map to return Assignedreplicas--Returns all replica objects under this partition Removereplica--Delete the given replica object Delete--Lock the way to delete a partition, including emptying all the replica objects under the partition, resetting the ISR collection object and emptying the leaderreplicaidopt, and deleting the logs under that partition Getleaderepoch--Get the value of Leaderepoch Makeleader--The current broker is elected as a leader for a given partition. First get all replica information for the partition and the leader, ISR collection, and Controller_epoch information for that partition. The second step is to create the corresponding replica instance (just the collection of replica numbers just received) and add it to assignedreplicamap. Thirdly, based on the set of ISR replica numbers, create an ISR replica instance set merge Delete those controllers have been removed, and then update Isr,leader and Zkversion and Leaderreplicaidopt, The new high-water-level metadata information is then created for this local copy, and the last reset of the log end displacement of the non-local copy is-1. If this partition belongs to the special topic of __consumer_offsets, it is also necessary to initiate an asynchronous request to read the partition and cache it, and finally return true to indicate that the election leader successful Makefollower--to elect a local copy as a follower, by placing leader and ISR empty. The process is still to get all the replica information under the partition and the partition's leader, ISR, and Controller_epoch information, create the corresponding copy instance and add it to the copy map, and then remove the copy that the controller has deleted. Finally, update the Isr,leader and zkversion information. If the partition belongs to a special offset topic, you also need to clean up the cache that belongs to that partition group. Finally, determine if the leader of this partition is not changed, return FALSE otherwise update leader returns TRUE. MAYBEINCREMENTLEADERHW--As the name implies, is pushing high water level, in fact, the previous code also has the concept of high water level--almost as in Oracle, is to mark the partition of the last committed message displacement. The high water level information is continuously passed to follower and is periodically updated to the disk's checkpoint file for broker recovery. This way, when a failed copy is restarted, it first recovers the latest high water level from the disk and discards all messages that are above the high water level. This is mainly because the message above the high water level is not guaranteed to be committed. After all this is done, the copy is re-added to the ISR and the system resumes. Okay, back to this method, it is worth noting that this method does not require lock protection, because all the outer methods that call it have already acquired the lock. The specific logic of this method is to first obtain the end displacement of all the replicas in the ISR, and extract the smallest one displacement as the new high water level, if the given leader copy of the high water level at the highest level of the natural nothing to do, otherwise Kafka need to update the high water level of this partition. Getoutofsyncreplicas--according to the given leader to select those behind leader too many copies, here too many refers to meet the lag time has been over the configured time or backward message number has been over configured number. UPDATEISR--Creates a new Leaderandisr instance based on the given replica collection, and then updates the zookeeper MAYBESHRINKISR--Remove too many copies of the backward leader from the ISR. Select the replicas to delete and then remove them from the ISR and update the ISR in the ZK and cache Appendmessagestoleader--Appends the message collection to the leader copy. First of all to determine whether the broker is leader, if not the error returned, otherwise get to the leader corresponding log and Min.insync.replicas configuration. If the current number of ISR is less than this configuration, then the message cannot be appended, and if these conditions are met, then the call to the Log.append method appends the message collection to the log and finally updates the location of the high water level equals--Two partitions if the topic or partition ID is different, the two partitions are considered hashcode--recalculate hash code based on partition ID toString--partition information, including topic, partition Id,leader,ar (assigned replicas), ISR Iv. Replica.scalaThe replica class is the replica object that represents the partition. We also follow the order of constructors, class fields, and class methods. 1. ConstructorsThe constructor receives 4 parameters, meaning: ①brokerid--broker id②partition--which partition the replica belongs to ③time--provide time-related services ④initialhighwatermarkvalue--initial high water level values ⑤log--log objects at the bottom of the replica object 2. Class Fields Highwatermarkmetadata-Logoffsetmetadata, which is created with the initial high water level value, represents the displacement value of the high water level. If the copy is not leader, then only the offset value is saved. Logendoffsetmetadata--Log End offset, all copies are saved. If it is a local copy, the value is the end shift of the log, and if it is a remote copy, the value will only be updated by follower. Initialize this value to-1 Logendoffsetupdatetimemsvalue--time stamp when updating the displacement Topic/partitionid--topic and partition to which the replicas belong 3. Class Methods isLocal--whether it is a local copy, mainly to see if the log object is empty, the local copy log object is not empty Logendoffset Setter--Update the end shift based on the given displacement value, and of course not updating the local log displacement (otherwise throwing the wrong)--specifically updating the values of the Logendoffsetmetadata and logendoffsetupdatetimemsvalue two variables Logendoffset Getter--if it is a local copy, returns the end offset of the local log directly (insert the address of the next message), otherwise returned by Logendoffsetmetadata Logendoffsetupdatetimems--Gets the time when offset was last updated Highwatermark Setter--Updates the high water level value of the local replica. Note that the high watermark value for a non-local replica under a partition cannot be updated Highwatermark Getter--Returns the Highwatermarkmetadata saved high water level offset value Converthwtolocaloffsetmetadata--all operations related to high water level can only be performed on local replicas, by invoking the Converttooffsetmetadata method of the copy's underlying log object to the high water level values saved by the incoming copy equals--If the topic, Broker ID, or partition of two replica objects is any different, it is considered to be two replica objects hashcode--Calculate hash code based on topic, Broker ID and partition

"Original" KAKFA cluster package source code Analysis