. Net business message queue and. net business queue

. Net business message queue and. net business queue

Open source QQ group:. net open source basic service 238543768

Open Source Address: http://git.oschina.net/chejiangyi/Dyd.BusinessMQ

# Business message queue ##
Business message queue is used for business decoupling and separation. It must be distributed, featuring high reliability, high performance, high real-time performance, high stability, and high scalability.

# Advantages :##
-A large number of business message accumulation capabilities
-No spof and Fault Monitoring and exception reminder
-Server Load balancer at the producer end, failover, automatic fault recovery, and concurrent message insertion.
-Consumer load balancing, fault persistence, automatic fault recovery, and concurrent message consumption.
-High message reliability, persistence, high performance, high real-time performance, high stability, and high scalability.
-Supports 99*99 message partitions. A single message partition supports storage of nearly 0.1 billion messages per day.
-When a consumer pulls messages in high concurrency and a large number of messages flood into the queue, as long as the consumption capability is sufficient, there will be no message delay. The more messages, the better the performance.

# Disadvantages :##
-Ensure that messages in the same partition are inserted sequentially (excluding network latency), but messages in different partitions may be in disorder.
-When messages are consumed in parallel or multiple partitions are consumed in parallel or load balancing, the order of messages consumed is out of order.

# Disadvantages :##
-Message load balancing is based on message partition storage, so messages in different partitions are in disordered order, but messages in the same partition are ordered.
-The Consumer load balancing of a message is also based on the partition of the message. When a single consumer subscribes to multiple partitions, it can also consume messages in parallel. Messages in different partitions are consumed in disorder, but messages in the same partition are consumed in order.

# Defect solution :##
-The Custom Server Load balancer algorithm of the producer is partitioned by business dimensions (users and merchants). Messages of multiple users can be in disordered order, and messages of a single user must be ordered ), different Dimensions can point to different partitions, but messages in a single dimension can be ordered.
-In the case of a fault, this solution will remove some faulty nodes, which means that the faulty node will immediately report an error (of course, you can also specify the faulty node for transfer, however, the transferred node messages will be consumed in advance, and The Faulty messages will be re-consumed after the fault is restored. In this case, messages will be consumed in disorder to the extent of failure ).
-This solution also has limitations on online seamless resizing and scalability performance. It depends on the specific load balancing algorithm, but generally, if you want to scale up, some messages will be migrated.

# Q & :##
### * 1. How to implement a large number of business message accumulation capabilities? *###
Each partition table supports about 0.1 billion message storage, and can be expanded by adding a partition table. The consumer only keeps the pointer of the last consumption in a partition for message consumption, so it does not affect the consumer.
Messages are persisted to the disk and do not reside in the memory. Theoretically, the memory is not affected.

### * 2. No spof and Fault Monitoring, error reminder? *###
Faults usually occur in redis, data nodes, management centers, and log centers.
Redis node failure will affect consumers' response to message consumption in a timely manner, generally with a delay of less than 5s. Does not affect message consumption speed and message consumption QPS
Data Node failure will affect the messages of producers and consumers, and cause temporary loss of messages (but they are all recoverable, depending on the degree to which the database is highly available ).
The producer can seamlessly remove the node, but reconnect again in 1 minute by default. The consumer continuously reports errors to the log, but does not affect the consumption of other partitions.
The management center failure will affect the heartbeat detection of producers and consumers and newly registered producers and consumers, but will not affect the specific message storage and sending and receiving of producers and consumers.
The log center failure does not affect producers and consumers, but affects log printing. The log center failure notifies the company's internal monitoring platform.
Although faults do not affect the running of existing online messages, performance problems and system stability still occur in the case of high concurrency. Therefore, once detected, it should be emphasized and handled in a timely manner.

### * 3. How can I configure Server Load balancer, failover, automatic fault recovery, and concurrent message insertion? *###
By default, Server Load balancer uses multiple partitions for sequential round-robin insertion. In the case of concurrency, round-robin insertion is performed concurrently in different partitions. If a data node fails, all partitions of the related data node are removed;
By default, the system reloads the faulty partition in one minute and retries.

### * 4. Consumer load balancing, fault persistence, automatic fault recovery, and concurrent message consumption. *###
By default, Server Load balancer on the consumer end is performed based on the shard subscribed by the consumer (a consumer can subscribe to Multiple shards, and multiple consumers with the same service can subscribe to multiple different shards for load ).
A consumer subscribes to multiple partitions. This consumer can enable parallel multi-partition consumption. Degree of parallelism = number of partitions, the best effect in theory.
If a partition node fails, the consumption of a single partition or data node is suspended and the log center is notified to print the error log. After the fault is restored, the consumption continues.

### * 5. High message reliability persistence, high performance, high real-time performance, high stability, and high stability. *###
After a message is transmitted to the message center, it is immediately persisted to the disk, so no message is lost. Producers can use multiple partitions for parallel insertion, and consumers can consume messages in parallel. Therefore, theoretically, the performance is scalable and unlimited.
Messages are obtained through pulling. messages are sent by redis for instant notification consumer pulling (instant messages are merged into redis notification messages within Ms by default ), generally, messages are consumed within 20 ms.
Compared with the push mode, the batch PULL mode provides better message sending performance when messages are pushed in high concurrency and a large number of messages are processed.
Stability is ensured based on database stability and failover. scalability reflects seamless online migration and resizing.

### * 6. Supports 9999 message partitions. A single message partition supports storage of nearly 0.1 billion messages per day. *###
The data node is 01 ~ 99, table partitions in the node are 01 ~ So it supports nearly 10 thousand partition nodes. The maximum number of mqids for a single table is (0.1 billion-1), which must meet general business requirements,
If not, you can scale up multiple partitions.

### * 7. When a consumer pulls messages in a pull mode and receives a large number of messages in high concurrency, as long as the consumption capability is sufficient, there will be no message delay. The more messages, the better the performance. *###
The push message mode ensures higher real-time performance. However, in the case of a large number of messages, message accumulation is more serious and performance may be affected.
The pull mode is slightly less efficient in ensuring real-time messaging. However, when a large number of messages flood in, batch pull is more efficient. In addition, the load of message distribution is transferred to multiple consumer terminals.

# Future improvement :##
1. leveldb will be used to rewrite storage in the future.
1. The Strip broker Service is used to support relatively reliable message services.
1. The message completes marking the local cache/persistence (or storing redis) and submits updates to the database every second to eliminate the bottleneck caused by frequent consumption.

# Architecture ##

# Demo ##

/// <Summary> /// send a message // </summary> /// <param name = "msg"> </param> public void SendMessageDemo (string msg) {// sending string example var p = ProducterPoolHelper. getPool (new BusinessMQConfig () {ManageConnectString = "server = 192.168.17.201; Initial Catalog = dyd_bs_MQ_manage; User ID = sa; Password = Xx ~! @ #; "}, // Management Center Database" dyd. mytest3 "); // queue path. the namespace of similar classes is the unique identifier of the queue. You must inform the O & M personnel to register in the message center in advance before using it. P. sendMessage (@ "1"); // sending object example/* var obj = new message2 {text = "text", num = 1}; var p = ProducterPoolHelper. getPool (new BusinessMQConfig () {ManageConnectString = "server = 192.168.17.237; Initial Catalog = dyd_bs_MQ_manage; User ID = sa; Password = Xx ~! @ #; "}, // Management Center Database" test. diayadian. obj "); // queue path. the namespace of similar classes is the unique identifier of the queue. You must inform the O & M personnel to register in the message center in advance before using it. P. sendMessage <message> (obj); */} private ConsumerProvider Consumer; /// <summary> /// receives the message // </summary> /// <param name = "action"> </param> public void ReceiveMessageDemo (Action <string> action) {if (Consumer = null) {Consumer = new ConsumerProvider (); Consumer. client = "dyd. mytest3.customer1 "; // clientid, the unique identifier of the message received (consumer). Once registered, it cannot be changed. After the service is deprecated, you must notify the O & M personnel to delete the consumer registration. Consumer. clientName = "client name"; // This is relatively casual and mainly used for self-identification. It must be a short Consumer. config = new BusinessMQConfig () {ManageConnectString = "server = 192.168.17.201; Initial Catalog = dyd_bs_MQ_manage; User ID = sa; Password = Xx ~! @ #; "}; Consumer. maxcompute emqthread = 1; // Number of threads for parallel processing. Generally, 1 is enough. If message processing is slow and you want to consume messages in parallel, consider the number of partitions being used = the number of parallel processing threads as the parallel efficiency is extremely optimal, but the cpu consumption should not be small. Consumer. MQPath = "dyd. mytest3 "; // The received queue must be correct Consumer. partitionIndexs = new List <int> () {1, 2, 3, 4, 5, 6, 7, 8}; // shard sequence number subscribed by the Consumer, starting from 1 to Consumer. registerReceiveMQListener <string> (r) => {/** when writing business code *, consider the failure of business processing. * 1. retry failed n times. * 2. If the retry fails, the message is marked as being processed. Then, skip the message processing and record this situation in other documents. * After a message is consumed, you must call MarkFinished to mark that the message has been consumed. */Action. invoke (r. objMsg); r. markFinished () ;}}}/// <summary> /// close the message subscription connection /// </summary> public void CloseReceiveMessage () {// register the consumer message. The consumer must disable the message after the program is closed ). Otherwise, an exception is terminated. You must manually wait for the connection to time out before registering again. If (Consumer! = Null) {Consumer. Dispose (); Consumer = null ;}}}

 

Part

 

Note :. net open-source message queues are few, especially for high-reliability message queues of businesses. net domain brings more solutions, more ideas and architecture design. At the same time, I also hope that people who know about MQ can give more suggestions and suggestions for this solution.

 

Author: Che jiangyi