Kafka Source code Analysis.

This records Kafka source code notes. (Code version is 0.8.2.1)

This is not the beginning of the Kafka boot sequence. Online has been a bunch of Kafka boot sequence and frames on the article. Here is no longer wordy, the main details of the Code Details section. The details will always be read and supplemented. If you want to see the Kafka framework and the boot sequence, the following link is recommended.

Http://www.cnblogs.com/davidwang456/p/5173486.html

The link author is labeled with a frame chart and a code. more clearly.

Configuration file parsing is implemented in Kafkaconfig.scala. This class file is in the server directory. The configuration file entries used below are provided by this class. This class inherits the Zkconfig class implementation. Zkconfig in the Utils.scala file. The function is to zookeeper The relevant parameters are separated out.

1, starting from Kafkascheduler.start () Start speaking.

Kafkascheduler is an instance object of the same name class. The class source file is in the Utils directory of the source code directory. The initialization of this object is in Kafkaserver The beginning of the class. The instantiation parameter is config.backgroundthreads. This parameter corresponds to the background.threads parameter in the configuration file. The default is 4.

The Kafkascheduler is encapsulated with Scheduldthreadpoolexecutor, The Stpe class is instantiated in the start function of this class. and set to no longer perform existing and recurring tasks when the actuator is closed. by setthreadfactory Set up its own factory function. The Newthread method in the Utils class in the Utils directory is called in the factory function. Doing some documentary work.

The Kafkascheduler class implements the scheduler method at the same time. This method encapsulates the scheduler method of Stpe. Used to pass the method that needs to be executed.

The shutdown function is also implemented by encapsulating the shutdown and Awaitshutdown methods of the STPE.

This object is passed to the Logmanager class through the Createlogmanager function. The Scheduler method is called in the Start method in this class to place the Logmanager task into the thread pool. Logmanager The specific task is log reservation, Log refresh, log check these three. The details of these three tasks will be explained in the Logmanager section.

2, then the Zkclient object was established through Zkinit ().

The Zkinit () function does this by initializing the Zkclient object with Zkclient and doing some Kafka in the Zookeeper path setting.

First check the CONFIG. Zkconnect whether the root path in the zookeeper is specified. Zkconnect is the Zookeeper.connect parameter in the configuration file. This parameter can be added after ip_list XXX This specifies the zookeeper working path. For example: 192.168.1.100,127.0.0.1:2181/kafka/dir, using substr directly in the Code (Index ('/')) The way to intercept chroot. If not set, use Zkutils.setupcommonpaths directly (zkclient) To create all the app directories for Kafka. The application directory can be found in the Zkutils.scala file. This file is in the Utils directory.

However, this function code is a little bit doubtful, this question is reserved for the time being. Or maybe I haven't read all the code.

PrivateDef INITZK (): Zkclient ={info ("Connecting to Zookeeper" +config.zkconnect)//General configuration Zookeeper.connect option is rarely followed by the path. But when followed by the path, the chroot will be the appended path. Val chroot= {      if(Config.zkConnect.indexOf ("/") > 0) config.zkConnect.substring (Config.zkConnect.indexOf ("/"))      Else""}//chroot is the time to append the path. The following code is executed. if(Chroot.length > 1) {//doubt in this place. Val zkconnforchrootcreation= config.zkConnect.substring (0, Config.zkConnect.indexOf ("/")//This is remove zookeeper IP list val zkclientforchrootcreation=Newzkclient (zkconnforchrootcreation, Config.zksessiontimeoutms, Config.zkconnectiontimeoutms, ZKStringSerializer      )//Then establish the Zkclient object.      Zkutils.makesurepersistentpathexists (zkclientforchrootcreation, chroot)//After this path is created by this method. Info ("Created Zookeeper Path" +chroot)//print a Create Info zkclientforchrootcreation.close ()//here to clean up the Zkclient object for this append path. }//after all these things. Normal use of Zkconnect to create a Zkclient object ... If chroot.length>1. This time zkconnect should be the same as in the If block need to extract IP list.
If there is no additional path at the beginning. There is no problem here. Val zkclient=Newzkclient (Config.zkconnect, Config.zksessiontimeoutms, Config.zkconnectiontimeoutms, ZKStringSerializer) Zkutils.setupcommonpaths (zkclient)//This function is calledzkutils.makesurepersistentpathexists function to establish the Kafka path.
Zkclient}

def setupcommonpaths (zkclient:zkclient) {//can see how this function works
For (Path <-Seq (Consumerspath, Brokeridspath, Brokertopicspath, Topicconfigchangespath, Topicconfigpath, Deletetopicspath))
Makesurepersistentpathexists (zkclient, path)//above these paths are defined in advance. defined as follows.
}

Val Consumerspath = "/consumers"//These paths are defined at the beginning of the Zkutils class. You can see in this definition whether or not you have added chroot. There is no effect.
Val Brokeridspath = "/brokers/ids"
Val Brokertopicspath = "/brokers/topics"
Val Topicconfigpath = "/config/topics"
Val Topicconfigchangespath = "/config/changes"
Val Controllerpath = "/controller"
Val Controllerepochpath = "/controller_epoch"
Val Reassignpartitionspath = "/admin/reassign_partitions"
Val Deletetopicspath = "/admin/delete_topics"
Val Preferredreplicaleaderelectionpath = "/admin/preferred_replica_election"

because it's version 0.8.2.1 's code, I went over the trunk version of the code and looked at it. This function has changed. Not directly calling the Zkclient implementation is called Zkutils implementation.

　　

　　

Kafka Source code Analysis.