Equeue file Persistence message key points of design ideas

There are three key data to persist

1. Message
2. queues, which hold message index information, which is the mapping information of the message's physical location in the file (Messageoffset) and the logical location (Queueoffset) in the queue;
3. Queue consumption progress, indicating that the current queue of messages consumed to the first few;

Design of sending messages

1. Producer sends binary data of the message to the broker;
2. What the broker does:
   • Single-threaded persistent message to memory-mapped file;
   • Puts the index information of the current message into a buffer, which can be implemented using Disruptor's Ringbuffer, single-threaded, and no lock.
   • A single thread reads the message index information from the buffer and writes the index information to the memory-mapped file;
   • The memory-mapped file of the message, the memory-mapped file of the message index are flushed to disk periodically, such as every 1s refresh, configurable;
3. When the broker puts the index information of the current message into the buffer, it returns immediately, and then producer receives the result of the message sent;

Other Notes:

1. Because it is not possible to use a file to save all the messages, it is certainly the way to use multiple files. That is, you use multiple files, whether you are saving a message or saving a message index. In addition, because there are multiple queues, each queue corresponds to more than one memory-mapped file. Directory naming rules for queue files: rootpath/topic/queueid/queue mapped files
2. When the broker puts the index information of the message into the buffer, it checks to see if the buffer reaches a certain level of water, such as the total size of the Ringbuffer 100W slots, if the water level is 80%, that is, when the current Ringbuffer in the available slots less than 20%, you should do flow control, Sleep 100s, for example, should not reach the water level theoretically, because writing a message index must be faster than writing the message itself;

Design of consumer News

1. Consumer tells the broker which message is currently required to pull the first position (Queueoffset) of which topic under which queue, and tells the maximum number of messages to be pulled;
2. The broker finds the corresponding queue according to topic and Queueid;
3. According to Queueoffset from the queue to get the message in the physical location of the file, that is, messageoffset;
4. Gets the message binary data from the memory-mapped file of the message according to Messageoffset;
5. Writes the message binary data to the temporary memory stream, which contains all the messages to be returned;
6. The binary data corresponding to the memory stream is returned to consumer when the number of messages is reached or no new message can be pulled;
7. Consumer parse the binary data to get all the message objects;

Broker timed cleanup of expired messages and message indexes

1. Every 10s scan has expired message file, expiration time can be configured, such as three days, scan, found that the last modification time of the file is 3 days ago, then deleted;
2. Every 10s scan has an expired message index file, determine whether the expiration is based on scanning each message index file, determine whether the last message index in the file Messageoffset is smaller than the smallest messageoffset, if small, This indicates that the message index file is meaningless and can be deleted;

Logic at broker startup

1. Scan all the messages on the disk to store the files and establish a memory map for each file;
2. Scans all the queues (message indexes) on the disk for storing files, establishing a memory map for each file;
3. For each queue, pre-recovers several files (such as the last 3 files) of the data into memory, remaining used to restore the time;
4. Similarly, for the file that stores the message, it also pre-recovers several (such as the last 3 files) to the memory; Most of the people who spend the most time are not too slow, they should always have caught up with the last three files;
5. The logic of the exception close broker, not yet thought out, but also need to think carefully;

Equeue file Persistence message key points of design ideas