JMS-based Active MQ message storage, jmsmq

JMS-based Active MQ message storage, jmsmq
I. Message Storage Methods

ActiveMQ supports persistent messages and non-persistent messages in JMS specifications.

• Persistent messages are usually used to ensure that messages are consumed by consumers regardless of whether the consumers are online or not. After the message is confirmed to be consumed, it will be deleted from the storage.
• Non-persistent messages are usually used to send notifications and real-time data. Generally, performance is required first, and message reliability is not required.
• MQ supports pluggable message storage, such as memory, files, and relational databases.

Storage of the Queue message model in ActiveMQ

The first-in-first-out (FIFO) mode is used for storage. A message can only be consumed by one consumer. A message is deleted only after it is confirmed to be consumed.

Topic message model (for persistent subscription)
The message obtained by each subscriber is actually a copy of the message. Only one copy of the message will be stored. MQ provides a pointer to direct the message storage and resends the message copy to the subscriber, the message cannot be deleted until all persistent subscribers are received.

Persistent storage:

Ii. Storage Method of KahaDB

KahaDB is the default persistent plug-in starting from ActiveMQ 5.4. The restoration time of KahaDb is much smaller than that of its predecessor AMQ and uses fewer data files. Therefore, it can completely replace AMQ. The persistence mechanism of kahaDB is also based on log files, indexing and caching.

(1) Main Features of KahaDB:

(2) applicable scenarios:

(3) configuration method conf/activemq. xml:

       <persistenceAdapter>            <kahaDB directory="${activemq.data}/kahadb"/>        </persistenceAdapter>

(4) Principle of KahaDB storage:

When an active consumer is used for temporary storage, the message is sent to the consumer and scheduled to be stored. if the message is confirmed in time, it does not need to be written to the disk. Data messages written to the disk are marked as deletable if the message is successfully sent in subsequent message activities. The system periodically clears or archives log files.

1. Internal Structure of KahaDB

Data logs: the message log contains the message log and some commands.
Cache: When an active consumer is in temporary storage, the message is sent to the consumer and scheduled to be stored. if the message is confirmed in time, it does not need to be written to the disk.
Btree indexes (message index): Used to reference the message id, which is stored in the memory and can be quickly located. MQ regularly saves the message index in memory to the metadata store to avoid occupying too much memory space when a large number of messages are not sent.
Redo log is used to maintain index integrity during abnormal shutdown.

2. directory structure:

 

Db log files: used to store messages (the default size is 32 MB). When the log is full, a new one will be created. When all messages in the log are deleted, this log file will be deleted or archived.
Archive directory: When the datalog is not archived by kahadb, it will be archived (controlled through the archiveDataLogs attribute ).
Db. data: stores Btree indexs.
Db. redo: stores redo files for restoring Btree indexs.

3. AMQ message storage

When a message is written, the message is written to the log file. The performance is high due to sequential append writing. To improve performance, create a message primary key index and provide a caching mechanism to further improve performance. The size of each log file is limited (32 MB by default ). When the size exceeds this value, the system creates a new file. When all messages are consumed, the system deletes the file or archives the file (depending on the configuration ). The main drawback is that AMQ Message creates an index for each Destination. If a large number of Queue is used, the size of the index file will occupy a lot of disk space. In addition, as the index is huge, the re-indexing speed will be very slow once the Broker crashes.

Features: Like KahaDB, it also contains transaction logs. Each destination contains an index file. AMQ is suitable for high-throughput application scenarios, but not suitable for scenarios with multiple queues.

Configuration method conf/activemq. xml:

<! -- AMQ directory: Data Storage path syncOnWrite: whether to synchronously write maxFileLength: Log File size --> <persistenceAdapter> <amqPersistenceAdapter directory = "$ {activemq. data}/AMQdb "syncOnWrite =" true "maxFileLength =" 10 mb "/> </persistenceAdapter>

1. Internal Structure of AMQ:

 

Data logs: the message log contains the Message Log
Cache: used for fast message Retrieval
Reference store indexes: Used to Reference messages in datalogs and associate them with message IDs.

2. directory structure:

Lock: ensure that only one borker accesses the file directory at a time
Temp-storag: used to store non-persistent messages (when not stored in memory), such as waiting for slow consumers to process messages
Kr-store: used to store referenced Message Log Data
Journal directory: contains message files, message logs, and message control information.
Archive: Archive data logs

Iv. JDBC Storage

Supports storing messages to relational databases through JDBC, which is inferior to file storage in performance and can query message information through relational databases.

Databases supported by MQ: Apache Derby, MYsql, PostgreSQL, Oracle, SQLServer, Sybase, Informix, and MaxDB.

Storage table structure:

A. ACTIVEMQ_MSGS: used to store messages. Both Queue and Topic are stored in this table:

• ID: auto-increment primary key of the database
• CONTAINER: Message Destination
• MSGID_PROD: The primary key of the message sender client.
• MSG_SEQ: the order in which messages are sent. MSGID_PROD + MSG_SEQ can constitute the MessageID of JMS.
• EXPIRATION: The EXPIRATION time of the message, which stores the number of milliseconds from to the present.
• MSG: the binary data of the Java serialized object of the message body.
• PRIORITY: PRIORITY, from 0-9. A greater value indicates a higher PRIORITY.

B. ACTIVEMQ_ACKS: used to store subscription relationships. If the Topic is persistent, the subscription relationship between the subscriber and the server is saved in this table:

• The main database fields are as follows:
• CONTAINER: Message Destination
• SUB_DEST: If a Static cluster is used, this field contains information about other systems in the cluster.
• CLIENT_ID: each subscriber must have a unique client ID to distinguish
• SUB_NAME: subscriber name
• SELECTOR: SELECTOR. You can select to consume only messages that meet the conditions. Conditions can be implemented using custom attributes, AND multi-attribute and or operations are supported.
• LAST_ACKED_ID: the ID of the consumed message.

C. ACTIVEMQ_LOCK (Message lock ensures that only one broker can access these table structures at a time ):
The activemq_lock table is useful only in the cluster environment. Only one Broker can obtain the message, called the Master Broker. Other brokers can only be used as backups when the Master Broker is unavailable. This table is used to record which Broker is the current Master Broker.

 

Configuration method:

1. Configure the data source conf/acticvemq. xml file:

<! -- Configure the data source --> <bean id = "mysql-ds" class = "org. apache. commons. dbcp. basicDataSource "destroy-method =" close "> <property name =" driverClassName "value =" com. mysql. jdbc. driver "/> <property name =" url "value =" jdbc: mysql: // localhost: 3306/activemq? RelaxAutoCommit = true "/> <property name =" username "value =" root "/> <property name =" password "value =" 111111 "/> <property name =" maxActive "value =" 200 "/> <property name =" poolPreparedStatements "value =" true "/> </bean>

2. Configure the persistenceAdapter in broke:

DataSource specifies whether the bean of the persistent database is created when createTablesOnStartup is started. The default value is true. In this way, the data table is created every time you start the database. This is generally set to true when you start the database for the first time, then change to false.

<! -- JDBC configuration --> <persistenceAdapter> <jdbcPersistenceAdapter dataSource = "# mysql-ds" createTablesOnStartup = "false"/> </persistenceAdapter>

Ps: the database activemq needs to be created manually.

5. Memory message storage

Memory message storage stores all persistent messages in the memory. You must pay attention to the JVM usage and memory restrictions. This applies to small messages with a small amount of data that can be consumed quickly, when MQ is disabled or goes down, unconsumed memory messages are cleared.

Set the broker property value persistent = "false" in configuration mode ":

    <broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.data}" persistent="false">