Kafka Source Depth Analysis-sequence 15-log file structure and flush brush disk mechanism

log file Structure

In front of us, we repeatedly talk about the concept of topic, partition, this article to analyze these different topic, different partition of the message, in the file, what structure is stored.

Interested friends can pay attention to the public number "the way of architecture and technique", get the latest articles.
or scan the following QR code:
each topic_partition corresponds to a directory

Suppose there is a topic called my_topic,3 partition, respectively, MY_TOPIC_0, My_topic_1, my_topic_2. There are 1 machine A in the first 2 parition, and the other paritition exist on another 1 machine B (here A, B refer to the leader machine corresponding to partition).

On machine A, there are 2 corresponding directories my_topic_0, my_topic_1, that is, each directory corresponds to a topic_partition, the directory name is topic_partition name.

In the directory, log files are rolled back by size or time, the file name is 00000000.kafka, 1200000.KAKFA ...

The file name is offset of the last 1 messages in the 1 log file. Here's 1200000.kafka, which means that there are 1.2 million messages before this file.

At the same time, there is a configuration parameter in the Kafka

Log.dir   //default is/tmp/kafka-logs

All the topic_partition directories are stored in this. file offset as message ID

As stated above, Kafka does not have an additional UUID-like way of generating a unique message ID for each message, but instead directly uses the message's offset in the file as the ID of that message.

Here's a key point: The offset here is not the physical location of the message in the file, but a sequential increment of the logical number. Starting from 0, one message per append, offset plus 1. Index File

As stated above, each message is incremented by a sequential logical number offset as its message Id. But the messages are all getting longer, so how do you find the file location where the message is located by its offset?

For example, to find the message offset = 95, the simplest way is to scan all the files sequentially, starting from the No. 0 record reading, and then read the location of offset = 95 where the message. But this approach is inefficient.

Kafka introduced the index file. Each of the 1. kakfa files has a corresponding. index file with exactly the same name as the 2 file. As shown in the following illustration:

Depending on the offset to find the file location, divided into 3 steps:
1th Step: Sort all the. kakfa file names by 2 points to get the. kakfa file
2nd step: In the corresponding. index file, 2 points to find the corresponding entry, that is, offset to position mapping.
3rd step: Get this position and navigate directly to the appropriate location in the. kakfa file. Then start with a position and scan sequentially to get the actual position.

The purpose of this. index file is to speed up the search. Even without this file, the 1th step got the. kakfa file, you can also find the location by scanning from scratch.

So this one. Index is a sparse index that stores the correspondence of 1 messages (offset, position) at every other range. variable-length message storage

We know that the news of Kafka is getting longer. Storage for variable-length records is typically at the front of the record, with a fixed number of bytes (such as 4 bytes) to store the length of the record.

Read this fixed 4 bytes, get the record length, and then read the later content according to the length.

As the following illustration shows, the Kafka record format is the same:

First 4 bytes, record length;
followed by a 1-byte, version number;
Next 4 bytes, CRC checksum value;
The last n bytes, the actual contents of the message.

Note: Different versions, the format will be slightly different. Flush Brush Disc mechanism

Familiar with the principles of the Linux operating system know that when we write data to the file system, the data is actually in the operating system page cache, and did not brush to disk up. If the operating system hangs at this point, the data is lost.

On the one hand, the application can call Fsync this system to force the brush disk, on the other hand, the operating system has a background thread, regular brush disk.

If an application calls a fsync every 1 times, the performance loss is significant, so it is generally a tradeoff between performance and reliability. As for an application, although the application hangs, as long as the operating system is not hanging, the data will not be lost.

In addition, Kafka is multi-copy after you have configured the synchronization replication. More than one copy of the data in the page cache, there are multiple copies of the probability of hanging off a copy, the probability is much smaller.

For Kafka, the relevant configuration parameters are also provided to allow you to trade-offs between performance and reliability:

Log.flush.interval.messages//  How many messages, brush Disk 1 times
log.flush.interval.ms  //Cut how long, brush Disc 1 times
log.flush.scheduler.interval.ms//Periodic brush disc, default 3000, i.e. 3s.

The default first 2 parameters are not set, that is, only the 3rd parameter function, that is, 3s clock brush disc 1 times. Multithreading writes the same log file

The

said earlier that the network model is 1+n+m, which means that the M worker thread may write the same 1 log files, which obviously need to be locked. The specific source code is relatively simple, here is not listed.