We continue to discuss the implementation principle of the console Consumer, this article focuses on the use of Zookeeperconsumerconnector, that is, all subsequent content from the following statement: Val connector = consumer.create (config) So that's the problem? What does it do after this statement? Let's look at the definition of Create method def create (config:consumerconfig): Consumerconnector = {val Consumerconnect = new Zookeeperconsumercon Nector (config) Consumerconnect} shows that all of its logic is to create a Zookeeperconsumerconnector instance and invoke its constructor. Now that the problem is simple, we have to figure out what Zookeeperconsumerconnector did when he created the instance: 1. The task of creating the Kafkascheduler----The scheduler is to commit the displacements to zookeeper 2 in a timed manner. Generates the consumer.id, in the format [group.id]_ hostname-timestamp-the first 8 bits of the random uuid, or if the consumer ID is specified on the command line, the format is [Group.id]_[consumer id]3. Create the connection Zookeeper client 4. Create a CONSUMERFETCHERMANAGER5. Generate timed tasks, according to the configuration of auto.commit.interval.ms to submit the displacement, the default is 1 minutes to submit the displacement to the zookeeper is to periodically save the consumed message displacement to zookeeper, the specific logic is very simple, This article does not dwell on here. We only care about the fourth step in the previous step-creating Consumerfetchermanager. So, what does Consumerfetchermanager do with it? As the name implies, Consumerfetchermanager is the manager of a consumer fetch thread, which maintains two mappings in memory: 1. The mapping of the fetch thread to the broker, that is, what the fetch thread 2 is on each broker. Topic Partition and partition consumption information mapping, where the partition consumption information contains a lot of content, such as the bottom of the consumption queue, saved to the ZK on the consumption of displacement, the maximum displacement obtained and the size of the information. With this information, a consumer thread manager makes it easy to redistribute consumer threads dynamically.
"Original" Kafka console consumer source code Analysis (ii)
