Messages and logs of roaming Kafka implementation

Address: http://blog.csdn.net/honglei915/article/details/37760631

Message format

A Message consists of a fixed-length header and a variable-length byte array. The header contains a version number and a CRC32 verification code.

/*** The format of a message with n Bytes is as follows ** if the version number is 0 ** 1. 1-Byte "magic" Mark ** 2. 4-byte CRC32 Verification Code ** 3. n-5 bytes ** if the version number is 1 ** 1. one-Byte "magic" Mark ** 2.1-byte parameters allow additional information such as whether the data is compressed, ** 3.4-byte CRC32 Verification Code ** of the decoding type. details of N-6 bytes **/

Logs

A topic named "my_topic" with two partitions. Its logs are composed of two folders,My_topic_0 and my_topic_1 each folder contains a specific data file, each data file is a series of log entities, each log entity has a four-byte integer n to mark the length of the message, followed by a message of n Bytes.Each message can be labeled with a 64-bit integer offset, indicating the start position of the message in the message stream sent to this partition. The name of each log file is the offset of the first log of this file. therefore, the name of the first log file is 00000000000. kafka. therefore, the name difference between each adjacent two files is a number S, and S is almost the maximum capacity of the log file specified in the configuration file.

The message format is maintained by a unified interface, so messages can be seamlessly transmitted between the producer, broker, and consumer. The message format stored on the hard disk is as follows:

Message length: 4 bytes (value: 1 + 4 + n) version: 1 bytecrc check code: 4 bytes specific message: n Bytes

Write operation

Messages are constantly appended to the end of the last log. When the log size reaches a specified value, a new file is generated. There are two parameters for the write operation. One specifies that when the number of messages reaches this value, the data must be refreshed to the hard disk, and the other specifies the time interval for refreshing to the hard disk, this guarantees data persistence. When the system crashes, only a certain number of messages or messages within a certain period of time will be lost.

Read operations

The read operation requires two parameters: a 64-bit offset and the maximum read volume of one second byte. S is usually larger than the size of a single message, but in some cases, S is smaller than the size of a single message. In this case, the read operation will retries continuously, and the read volume will be doubled each retry until a complete message is read. You can configure the maximum value for a single message so that the server rejects messages whose size exceeds this value. You can also specify a maximum Read Attempt limit for the client to avoid infinite retries to read a complete message.

In actual read operations, you must first locate the log file where the data is located, and then calculate the offset in the log based on the offset (the offset in front is the offset of the entire partition ), then read at the offset position. The location operation is completed by the binary search method. Kafka maintains the offset range for each file in the memory.

The format of the result sent to the consumer is as follows:

MessageSetSend (fetch result)total length     : 4 byteserror code       : 2 bytesmessage 1        : x bytes...message n        : x bytes

MultiMessageSetSend (multiFetch result)total length       : 4 byteserror code         : 2 bytesmessageSetSend 1...messageSetSend n

Delete

The log Manager allows you to customize deletion policies. The current policy is to delete the logs modified before n days (delete by time). You can also use another policy: retain the last n GB of data (delete by size ). To avoid blocking read operations during deletion, the copy-on-write mode is adopted, the binary search function of the read operation is actually performed on a static snapshot copy, which is similar to the copyonwritearraylist of Java.

Reliability Assurance

The log file has a configurable parameter M. messages that exceed the cached value will be forcibly refreshed to the hard disk. A log correction thread cyclically checks the messages in the latest log file to confirm that each message is valid. The valid standard is: the size and maximum offset of all files are smaller than the size of the log file, and the CRC32 verification code of the message is consistent with the verification code stored in the message entity. If an Invalid Message is found at an offset, the content from this offset to the next legal offset will be removed.

There are two situations that must be considered: 1. When a crash occurs, some data blocks cannot be written. 2. Some blank data blocks are written. The second cause is that each file has an inode (inode refers to a Data Structure in many "UNIX-like file systems. Each inode stores a file system object in the file system, including files, directories, sizes, device files, sockets, pipelines, and so on), but it cannot guarantee the order of updating inode and writing data, when the size information stored by inode is updated but data is written, a crash occurs, leading to a blank data block. The CRC check code can check these blocks and remove them. Of course, data blocks that are not written due to crashes will be lost.