Metaq High-availability configuration (asynchronous replication and synchronous replication)

Introduced

One important feature of Meta compared to Kafka is the implementation of the message high availability scenario, which we call the HA scenario. The message is written to the disk immediately after it is sent to the broker and returned to the client to tell the producer that the message was sent successfully, and the control of the unflushThreshold unflushInterval two parameters can guarantee the security of the stand-alone message data, as long as the machine's disk is not permanently damaged. Messages can always be restored and delivered to consumers after a reboot. However, if you encounter permanent damage to the disk or permanent corruption of the data file, the message data on the broker may be permanently lost. To prevent this from happening, a viable solution is to copy the message data to multiple machines, similar to the master-slave replication feature of MySQL.

Synchronous replication and asynchronous replication

Meta provides asynchronous replication and synchronization functions like MySQL master-slave replication, which correspond to different levels of reliability. In theory, synchronous replication leads to higher levels of reliability, and asynchronous replication can lose a very small amount of message data due to the presence of delays, which, in turn, can result in a loss of performance because the write succeeds when it is written to two or more broker machines synchronously.

In practice, I prefer the architecture of asynchronous replication because the architecture of asynchronous replication is relatively simple and easy to maintain and restore, and has no impact on performance. Synchronous replication is relatively high in operation and maintenance, the mechanism is complex and error-prone, and fault recovery is more troublesome. asynchronous replication coupled with disk-made disk arrays is sufficient to handle very demanding data reliability requirements.

Asynchronous Replication Configuration

Suppose you have configured a broker server based on how you started this document and configured a topic for test , now you want test to replicate to another slave broker to ensure high availability of the message data. You can do this:

1. First, you need to deploy a new broker, still referencing how to start the document, and configure Server.ini to copy from Master Broker.

2. Second, configure the slave file. Edit conf/async_slave.properties:

#slave编号,大于等于0表示作为slave启动,同一个master下的slave编号应该设不同值.slaveId=0#作为slave启动时向master订阅消息的group,如果没配置则默认为meta-slave-group#不同的slaveId请使用不同的groupslaveGroup=meta-slave-group#slave数据同步的最大延时,单位毫秒  slaveMaxDelayInMills=500#是否自动从master同步server.ini, 1.4.2新增选项#第一次仍然需要自己拷贝server.ini，后续可以通过设置此选项为true来自动同步autoSyncMasterConfig=true

The meaning of configuration parameters please read the comments yourself. Visible, a master can be copied to multiple slave.

3. Execute the following command to start slave:

bin/metaServer.sh start slave

4. First copy because it takes time to fully synchronize with master, you can observe replication in the data file's directory.

5. Note that asynchronous replication of slave will participate in consumer activity, message consumers can get messages from slave and consume, and consumers will randomly pick one from master and slaves as a consumer broker.

6. Note that starting with 1.4.2, the Autosyncmasterconfig option allows you to configure whether to automatically synchronize Master's Server.ini to the slave of asynchronous replication when Master's Server.ini file is changed and passed bin/metaServer.sh reloadafter that, slave will monitor this change and automatically synchronize.

Limitations of asynchronous replication

• Asynchronous replication has a delay, although it can be set slaveMaxDelayInMills to control the delay.

Fault Handling for asynchronous replication

• Master permanent failure: Start slave as master and remove slave from the startup parameters, i.e.metaServer.sh restart

• Slave permanent failure: Start a new broker and configure the new slave to start as master.

Synchronous Replication Configuration

Todo

Metaq High-availability configuration (asynchronous replication and synchronous replication)