The broker internal architecture of the Apache Kafka series

The main submodules of Kafka broker are described below to help you better learn and understand Kafka source code and architecture.

Several sub-modules are described as follows:

Kafka API layer Logmanager and Log replicamanager zookeeperconsumerconnector service Schedule is how several modules of the system are composed into a schema diagram:
startup process
Zkclient Initialize =>logmanager start () = Socketserver start () =>requesthandlerpool initialize =>replicamanager start () = >
Kafkacontroller start () =>topicconfigmanager start () = Kafkahealthcheck start ()
Core IO processing of broker in Requesthandlerpool

Zkclient initialization
Using the link between the initial and ZK of the third party zkclient, Zkclient is responsible for maintaining the connection,session with ZK.

Logmanager initialization
All read and write requests will instantiate their own log instance objects. At the same time, the log thread in the background will be responsible for maintaining partitions and log segment.

Socketserver initialization
NIO Socket Server
* 1 acceptor thread is responsible for receiving requests and transferring them to processor threads for processing
* N Processor Threads is responsible for the actual processing of each socket request


• M Handler Threads is responsible for the event mechanism that urges each processor request to reply to the message of the product request.

Replicamanager initialization
• Manage the partition copy of the broker

Kafkacontroller initialization
Management of the Controller State responsible for the broker

Topicconfigmanager initialization
The state management work of the topic responsible for the broker

Kafkahealthcheck initialization
Responsible for the status of the broker in ZK's maintenance work
Requesthandlerpool initialization
Core IO processing center, handled through event-driven mode

Maintain 2 queues,
One is the request queue
One is the response queue
By, Kafkaapis invokes (about view Kafkaapis) The event type handler, and handles the related event.
Requested event types Total Producekey, Fetchkey, Offsetskey, Metadatakey, Leaderandisrkey, Stopreplicakey,
Updatemetadatakey, Controlledshutdownkey, Offsetcommitkey, Offsetfetchkey
Please specify reproduced from: http://blog.csdn.net/lizhitao/article/details/38352947