Jafka source code analysis-logmanager

In Kafka, logmanager manages all the logs on the broker (each topic-partition is a log ). By reading the source code, we can see that the specific functions are as follows:

1. Clear the Message Queue according to the preset rules.

2. Perform the flush operation on the Message Queue according to the preset rules ).

3. Connect zookeeper to perform zookeeper operations related to broker, topic, and partition.

4. Manage all the logs on the broker.

The following describes the implementation of these functions in detail.

1. Log Management

Logmanager contains the member variable logs. The key of logs is topic, and the value is pool <integer, log> (this value is a map, the primary key is partition, and the value is the log corresponding to this partition ). Therefore, logmanager stores all message queues on the broker through logs.

private final Pool<String, Pool<Integer, Log>> logs = new Pool<String, Pool<Integer, Log>>();

After initialization, logmanager needs to traverse the Message Queue root directory configured in the configuration file. Search for and generate logs through traversal. The specific implementation of this traversal is in the method load:

① Obtain all the files in the message queue root directory

② Perform the following operations on each file in the root directory:

1. If it is a directory, it may be a log, otherwise it is not ignored

2. For the file name analysis through the directory 1, the directory file name consists of two parts: Topic-partition

3. for directories that pass through 2, use the directory, resolved topic, and parsed partition to generate a log

4. Add the logs generated by 3 to the logs log pool.

5. Finally, determine the partition size configured in the directory resolution and configuration file. If the configuration file is small, update the configuration

Ii. Message Queue cleaning

Message Queue cleanup is periodically called by scheduler. The specific call is implemented in the load function. The main deletion is implemented in the cleanlogs function. Message Queue cleanup can be divided into two situations: one is to delete messages when the preset time is exceeded, and the other is to delete messages when the preset size is exceeded, which correspond to two functions: cleanupexpiredsegments and cleanupsegmentstomaintainsize. The first case is relatively simple, because each segment corresponds to a file, and the lastmodifiedtime of the file is compared with the current time of the system to determine whether it has timed out. If it times out, It is deleted. In the second case, first compare the log size and configuration size. If the value is smaller than the configured size, it is not deleted. If the value is greater than the configured size, the length exceeds the configured size (set as the difference value ); then delete the segment smaller than the difference value (this is a bit confusing, so that the deletion will not delete some of the latest message queues ).

 if (this.scheduler != null) { this.scheduler.scheduleWithRate(new Runnable() {                public void run() {                    try {                        cleanupLogs();                    } catch (IOException e) {                        logger.error("cleanup log failed.", e);                    }                }            }, 60 * 1000, logCleanupIntervalMs);}

Iii. Message Queue persistence

The flush operation on the message queue is also completed by a separate thread. This thread compares the last flush time of log with the current system time to determine whether flush is required. If so, the thread persists to the file. Note: The persistence of the Message Queue also determines when a new message is added. If the number of new messages saved by a log exceeds the preset value, the flush operation is performed.

In Kafka, logmanager manages all the logs on the broker (each topic-partition is a log ). By reading the source code, we can see that the specific functions are as follows:

1. Clear the Message Queue according to the preset rules.

2. Perform the flush operation on the Message Queue according to the preset rules ).

3. Connect zookeeper to perform zookeeper operations related to broker, topic, and partition.

4. Manage all the logs on the broker.

The following describes the implementation of these functions in detail.

1. Log Management

Logmanager contains the member variable logs. The key of logs is topic, and the value is pool <integer, log> (this value is a map, the primary key is partition, and the value is the log corresponding to this partition ). Therefore, logmanager stores all message queues on the broker through logs.

private final Pool<String, Pool<Integer, Log>> logs = new Pool<String, Pool<Integer, Log>>();

After initialization, logmanager needs to traverse the Message Queue root directory configured in the configuration file. Search for and generate logs through traversal. The specific implementation of this traversal is in the method load:

① Obtain all the files in the message queue root directory

② Perform the following operations on each file in the root directory:

1. If it is a directory, it may be a log, otherwise it is not ignored

2. For the file name analysis through the directory 1, the directory file name consists of two parts: Topic-partition

3. for directories that pass through 2, use the directory, resolved topic, and parsed partition to generate a log

4. Add the logs generated by 3 to the logs log pool.

5. Finally, determine the partition size configured in the directory resolution and configuration file. If the configuration file is small, update the configuration

Ii. Message Queue cleaning

Message Queue cleanup is periodically called by scheduler. The specific call is implemented in the load function. The main deletion is implemented in the cleanlogs function. Message Queue cleanup can be divided into two situations: one is to delete messages when the preset time is exceeded, and the other is to delete messages when the preset size is exceeded, which correspond to two functions: cleanupexpiredsegments and cleanupsegmentstomaintainsize. The first case is relatively simple, because each segment corresponds to a file, and the lastmodifiedtime of the file is compared with the current time of the system to determine whether it has timed out. If it times out, It is deleted. In the second case, first compare the log size and configuration size. If the value is smaller than the configured size, it is not deleted. If the value is greater than the configured size, the length exceeds the configured size (set as the difference value ); then delete the segment smaller than the difference value (this is a bit confusing, so that the deletion will not delete some of the latest message queues ).

 if (this.scheduler != null) { this.scheduler.scheduleWithRate(new Runnable() {                public void run() {                    try {                        cleanupLogs();                    } catch (IOException e) {                        logger.error("cleanup log failed.", e);                    }                }            }, 60 * 1000, logCleanupIntervalMs);}

Iii. Message Queue persistence

The flush operation on the message queue is also completed by a separate thread. This thread compares the last flush time of log with the current system time to determine whether flush is required. If so, the thread persists to the file. Note: The persistence of the Message Queue also determines when a new message is added. If the number of new messages saved by a log exceeds the preset value, the flush operation is performed.