RABBITMQ persistence mechanism

I've already written an article about RABBITMQ persistence, but that article focuses on the code-level writing process, and it's not clear when the persistence operation occurs and when it's flushed to disk, and this article focuses on these issues.

When do messages need to be persisted?

According to the official blog post, RABBITMQ writes the message to disk in two cases:

1. The message itself requires the message to be written to disk when publish;
2. Memory tight, need to transfer some in-memory messages to disk;

When will the message be brushed to disk?

1. There will be a buffer before writing the file, the size of 1M (1048576), the data is written to the file, the first will be written to this buffer, if the buffer is full, the buffer will be written to the file (not necessarily brush to disk);
2. There is a fixed brush disk Time: 25MS, that is, regardless of buffer full of dissatisfaction, every 25ms,buffer data and not flushed to the disk of the file content will certainly be brushed to disk;
3. After each message is written, if there is no subsequent write request, the written message is directly brushed to disk: Implemented using Erlang's receive x after X., as long as there is no message in the process's mailbox, a timeout message is generated, and timeout triggers the brush disk operation.

The format of the message in the disk file

Messages are saved in the $mnesia/msg_store_persistent/x.rdq file, where x is a numeric number, starting with 1, each file is up to 16M (16777216), and more than that size generates a new file with a file number plus 1. The message exists in the file in the following format:

<<size:64, Msgid:16/binary, msgbody>>

The guid,msgbody generated by MsgId for RABBITMQ through Rabbit_guid:gen () contains messages corresponding to the Exchange,routing_keys, the contents of the message, the protocol version of the message, Message content format (binary or other) and so on.

When is the file deleted?

When the proportion of spam messages (messages that have been deleted) in all files is greater than the threshold (garbage_fraction = 0.5), a file merge operation (at least three files exist) is triggered to improve disk utilization.

Publish the message when the content is written, the ACK message is removed when the content is deleted (update the file's useful data size), and when a file's useful data equals 0 o'clock, the file is deleted.

When does the message index need to be persisted?

The persistence of an index is similar to the persistence of a message and is written to disk in two cases: either it needs to be persisted or it needs to be freed due to memory constraints.

When will the message index be brushed to disk?

1. There is a fixed brush disk Time: 25ms, the index file content must be brushed to disk;
2. After each message (and index) is written, if there is no subsequent write request, the written index is directly brushed to disk, and the implementation is consistent with the timeout brush disk for the message.

Format of the message index in the disk file

Each queue maintains an index on the messages in the queue, one message per incoming queue, an index plus 1, and a file (Segment) in 2^14 (16384) entry as the index is persisted. When the index is written, it may not be written sequentially, and in order to avoid excessive disk addressing, the index information is first saved in the journal file, and when the number of entry in the file reaches 65,536, the contents are written to the segment file. Where journal is stored in the $MNESIA/QUEUES/MD5 (queuename)/journal.jif file, the index is saved in $MNESIA/QUEUES/MD5 (queuename)/ X.idx file, where x is a numeric number, similar to the number of a message file.

The index information is formatted in the segment file in two ways:

1. Corresponding publish messages: <<1:1, Ispersistent:1, relseq:14, Msgid:16/binary, expiry:8/binary>>;
2. Corresponds to deliver or ACK message: <<01:2, relseq:14>>.

When is the file deleted?

The Rabbit_msg_index module maintains a unacked count for each segment, one message per publish plus 1, one message per ack minus 1, and when Unacked=0, the file is deleted.

Note: The size of the persisted file can be modified by configuration parameters Msgstore_file_size_limit, the maximum number of entry in the journal file can be queue_index_max journal_entries configuration by parameters, see here

Finally, share with our partners

see a degree hundred do in Do promotion, a penny to receive gloves, winter, everyone speed , chain contact this ~ ~ ~

The picture is as follows:

RABBITMQ persistence mechanism