ACTIVEMQ message Persistence mechanism

In order to avoid the loss of information after an unexpected outage, it is necessary to restore Message Queuing after a reboot, and the message system will generally adopt a persistence mechanism.

ACTIVEMQ's message persistence mechanism has JDBC,AMQ,KAHADB and leveldb, and regardless of which persistence method is used, the message's storage logic is consistent.

After the sender sends the message out, the message center first stores the message to the local data file, the memory database, or the remote database, and then attempts to send the message to the receiver, which then deletes the message from the store and fails to continue.

After the message center is started, first check the established storage location, if you have not sent a successful message, you need to send the message.

>> jdbc Persistence mode

Using the JDBC persistence method, the database creates 3 tables: Activemq_msgs,activemq_acks and Activemq_lock.
ACTIVEMQ_MSGS is used to store messages, and queue and topic are stored in this table.

(1) Configuration method

The way to configure persistence is to modify the Conf/acticvemq.xml file in the installation directory,

First define a mysql-ds MySQL data source, then configure the jdbcpersistenceadapter in the Persistenceadapter node and reference the data source you just defined.

<persistenceAdapter>      <jdbcpersistenceadapter datasource= "#mysql-ds" createtablesonstartup= "false"/ >  </persistenceAdapter>  

DataSource Specifies whether the bean,createtablesonstartup of the persisted database creates a data table at startup, and the default value is true, so that every boot will create a data table, typically set to true the first time it is started, It is then changed to false.
To configure JDBC persistence using MySQL:

<beans>    <broker brokername= "Test-broker" persistent= "true" xmlns= "http://activemq.apache.org/schema/ Core ">        <persistenceAdapter>            <jdbcpersistenceadapter datasource=" #mysql-ds " Createtablesonstartup= "false"/>         </persistenceAdapter>    </broker>    <bean id= " Mysql-ds "class=" Org.apache.commons.dbcp.BasicDataSource "destroy-method=" Close ">        <property name=" Driverclassname "value=" Com.mysql.jdbc.Driver "/>         <property name=" url "value=" jdbc:mysql://localhost/ Activemq?relaxautocommit=true "/>         <property name=" username "value=" activemq "/>         <property name= "Password" value= "Activemq"/>         <property name= "maxactive" value= "$"/> <property name=         " Poolpreparedstatements "value=" true "/>     </bean></beans>

　

(2) database table information
ACTIVEMQ_MSGS is used to store messages, and queue and topic are stored in this table:
ID: Self-growing database primary key
CONTAINER: The destination of the message
Msgid_prod: Primary key for the message sender client
MSG_SEQ: is the order in which messages are sent, MSGID_PROD+MSG_SEQ can make up the JMS MessageID
Expiration: The expiration of the message, storing the number of milliseconds from 1970-01-01 to the present
MSG: Binary data for the Java serialized object of the message ontology
Priority: Precedence, from 0-9, the higher the value, the greater the priority

Activemq_acks is used to store subscription relationships. In the case of persistent topic, the Subscriber and server subscription relationships are saved in this table:
The main database fields are as follows:
CONTAINER: The destination of the message
Sub_dest: If you are using a static cluster, this field will have information about other systems in the cluster
CLIENT_ID: Each Subscriber must have a unique client ID to differentiate
Sub_name: Subscriber Name
SELECTOR: Selector, you can choose to consume only messages that meet the criteria. Conditions can be implemented with custom attributes that support multiple attributes and and or operations
LAST_ACKED_ID: Records the ID of the consumed message.

Table Activemq_lock is useful in a clustered environment where only one broker can get the message, called Master Broker,
The other can only be used as a backup to wait for master broker to be unavailable before it becomes the next master broker.
This table is used to record which broker is the current master broker.

>> AMQ Way

Performance is higher than JDBC, when writing a message, the message is written to the log file, which is highly performance due to sequential append writes. To improve performance, create a message primary key index and provide a caching mechanism to further improve performance. There is a limit to the size of each log file (default 32m, which can be configured by itself).
When this size is exceeded, the system will re-establish a file. When all messages are consumed, the system will delete the file or archive (depending on the configuration).
The main drawback is that AMQ message creates an index for each destination, and if a large number of queues are used, the size of the index file consumes a lot of disk space.
And because the index is huge, rebuilding the index can be very slow once the broker crashes.

The configuration fragment is as follows:

<persistenceAdapter>     <amqpersistenceadapter directory= "${activemq.data}/activemq-data" Maxfilelength= "32MB"/></persistenceadapter>

Although the AMQ performance is slightly higher than that of the kaha db below, it is deprecated because of the long time it takes to rebuild the index and the size of the index file that is too large for disk space.

>> kahadb Way

KAHADB is the default persistence plugin starting from ACTIVEMQ 5.4, and is the persistence that our project now uses.

KAHADB recovery time is much smaller than its predecessor AMQ and uses less data files, so it can replace AMQ completely.
The persistence mechanism of KAHADB is also based on log files, indexes, and caches.

How to configure:

<persistenceAdapter>     <kahadb directory= "${activemq.data}/activemq-data" journalmaxfilelength= "16MB" /> </persistenceadapter>directory: Specifies the storage directory for persistent messages Journalmaxfilelength: Specifies the log file size of the saved message, depending on your actual application configuration

(1) KAHADB main features
1, the log form to store messages;
2, the message index to B-tree structure storage, can be updated quickly;
3, fully support JMS transactions;
4, support a variety of recovery mechanisms;

(2) Structure of KAHADB

Messages are stored in a file-based data log. If the message is sent successfully, the change is marked as removable. The system periodically clears or archives the log files.
The location index of the message file is stored in memory so that it can be quickly located. Periodically save the in-memory message index to the metadata store, avoiding the message index consuming too much memory space when a large number of messages are not sent.

Data logs:
Data logs is used to store message logs, and the entire contents of the message are in data logs.
As with AMQ, a data logs file is larger than the specified maximum value, and a new file is created. Also the end of the file is appended, writing performance is fast.
Each message has a count reference in the data logs, so when all the messages in a file are not needed, the system automatically deletes the files or puts them in the archive folder.

Metadata Cache:
Caches the messages that are used to store online consumers. If consumers are already spending quickly, then these messages don't need to be written to disk anymore.
The Btree index creates an index based on MessageID, which is used to quickly find messages. The index also maintains a persistent subscriber-to-destination relationship, as well as a pointer to each consumer consumer message.

Metadata Store
The metadata for messages in the message log is saved in the Db.data file, and is stored in the b-tree structure, and the data is updated periodically from the metadata cache. Some information that exists in the message log is also backed up in the Metadata store, which allows the broker instance to start quickly.
Even if the metadata store file is corrupted or deleted by mistake. The broker can read the data logs and recover it, but the speed will be relatively slow.

>>leveldb Way

Since the ACTIVEMQ 5.6 release, the LEVELDB persistence engine has been introduced.
The default persistence mode is still kahadb, but leveldb persistence performance is higher than kahadb, which can be a later trend.
The ACTIVEMQ 5.9 version provides data replication based on LEVELDB and zookeeper, a Preferred data replication scheme for Master-slave methods.

Organized from
Message persistence in ActiveMQ (i)

ACTIVEMQ Learning Note 03-Message Persistence

ACTIVEMQ message Persistence mechanism