. NET Business Message Queuing

Open source QQ group:. NET open source basic services 238543768

Open Source Address: Http://git.oschina.net/chejiangyi/Dyd.BusinessMQ

# # Business Message Queue # #
Business Message Queuing is a decoupled and decoupled business that should be distributed, high reliability, high performance, high real-time, high stability, and high scalability.

# # Advantages: # #
-Large number of business message stacking capabilities
-no single point of failure and fault monitoring, abnormal alerts
-Terminal load balancing, failover, fault auto-recovery, parallel message insertion.
-consumer-side load balancing, fault retention, automatic failure recovery, parallel messaging consumption.
-Message high reliability persistence, higher performance, high real-time, high stability, high scalability.
-Supports 99*99 message partition, single message partition supports nearly 100 million message storage per day.
-Consumer pull way to get the message, in the case of high concurrency, a large influx of messages, as long as the consumption capacity enough, there will be no message delay, the more the better the performance of the message.

# # Cons: # #
-The message sequence is guaranteed to be inserted, so that messages of the same partition are sequential (excluding network latency), but the number of partitions may be out of order.
-In the case of concurrent consumption of messages or multiple partitions in parallel consumption or load balancing, the order of message consumption is chaotic.

# # Defect Reason: # #
-Message load Balancing is a message-based partition store, so messages between multiple partitions are scrambled, but messages of the same partition are ordered.
-The consumer load balancer for messages is also balanced based on the message partition, while a single consumer subscribes to multiple partitions and can also consume in parallel. means that the consumption of messages in different partitions is out of order, but the consumption of messages in the same partition is sequential.

# # Disadvantage Solution: # #
-Producer Custom load balancing algorithms that are partitioned according to business dimensions (user, merchant), etc. (multiple users can message out of order, messages for individual users must be sequential), different dimensions can point to different partitions, but messages from a single dimension can be guaranteed in order.
-This solution in the case of failure, the fault will remove some fault nodes, which means that the fault node will immediately error (of course, you can also specify the fault node to transfer, but the transferred node messages will be consumed in advance, the failure of the message will be re-consumption after recovery failure, so that the failure level of the message disorderly consumption).
-There will also be limitations on the ability to seamlessly scale and extend the solution on-line, depending on the load-balancing algorithm, but in general, if you want to expand, you will still have partial message migrations.

# # Quiz: # #
# # # # Number of Business message stacking capabilities, how to achieve? * ###
Each partition table supports about 100 million of the message store and can be scaled by increasing the partition table. Consumers consume messages, keeping a pointer to the last consumption of a partition, so they don't affect consumers.
Messages persist to disk, do not reside in memory, and theoretically do not affect memory.

No single point of failure and fault monitoring, abnormal reminders? * ###
Failures typically occur in Redis, data node, Admin Center, and log Center.
Redis node failures can affect consumer response time for message consumption, typically within 5s of latency. Does not affect message consumption speed and message consumption QPS
Data node failures affect producers and consumers ' messages and cause messages to be temporarily lost (but are recoverable, depending on how highly available The database is).
The producer side will seamlessly remove the nodes, but will retry the reconnect by default for 1 minutes. The consumer will continue to error to the log, but will not affect other partition consumption.
Management center failures can affect producers and consumers of heartbeat detection and newly registered producers, consumers, but will not affect the producer and consumer specific message store and send receive.
Log Center failures do not affect producers and consumers, but they affect the printing of logs, and log center failures notify the company's internal monitoring platform.
Although the fault does not affect the messages that are already running on the line, there are performance issues and system stability in high concurrency, so once you find that you need to pay attention and deal with them in a timely manner.

The producer side load balancing, failover, fault automatic recovery, parallel message insertion? * ###
The default load balancer uses multiple partitioned sequential polling inserts, in which case the insert is inserted in parallel to a different partition, and a data node fails, removing all the partitions of the relevant data node;
The default 1 minutes will reload the failed partition for retry.

Customer-side load balancing, failure to maintain, automatic failure recovery, parallel message consumption. * ###
The default consumer-side load balancer is based on the partitioning of the consumer subscription (a consumer can subscribe to multiple partitions, and multiple consumers of the same business can subscribe to multiple different partitions for load).
A consumer subscribes to multiple partitions, which the consumer can turn on in parallel for multi-partition consumption. The degree of parallelism = number of partitions, the best effect theoretically.
If a partition node fails, a single partition or data node pauses consumption and notifies the log Center to print the error log. After the failure is resumed, consumption continues.

# # # The message is high reliability persistence, higher performance, high real-time, high stability, high stability. * ###
Once the message is delivered to the message center, it is persisted to disk immediately, so no message is lost. Producers can use multiple partitions for parallel insertion, consumers can use parallel message consumption, so the theoretical performance is extensible unlimited amount.
Messages are obtained by pulling the message, which is immediately notified to the consumer by Redis (Instant Messaging is merged into the REDIS notification message by default in 500ms), and the message is typically consumed within 20ms.
How to bulk pull messages with a message push in the case of high concurrency and large amounts of message processing, message delivery performance should be better.
Stability is based on the stability of the database and the level of failover to ensure that the scalability of the online seamless migration and expansion.

# # # *6.9,999 message partitions are supported, and a single message partition supports nearly 100 million message stores per day. * ###
The data node is 01~99, the table partition inside the node is 01~99, so it can support nearly 10,000 partition nodes. The mqid of a single table should be (100 million-1) and should meet the general business needs,
If not, it can be expanded by multiple partitions.

# # # *7. Consumer pull way to get the message, in the case of high concurrency, a large influx of messages, as long as the consumption capacity is sufficient, there will be no message delay, the more the better the performance of the message. * ###
The push-push message pattern guarantees higher real-time performance, but in the case of a large number of messages, the message is piled up more severely, and there is an impact.
The pattern of pull messages is slightly worse at guaranteeing the real-time nature of the messages, but in the case of a large influx of messages, the bulk pull messages are more efficient. It also transfers the load of message distribution to multiple consumers.

# # Future improvements: # #
1. Future use of LEVELDB rewrite storage.
1. The Peel Broker service is used to support relatively reliable messaging services.
1. Message Completion flag local cache/persistence (or storage redis), commit updates per second to the database, eliminating bottlenecks caused by frequent consumption.

# # Architecture # #

# # using Demo Example # #

 /// <summary>        ///Send Message/// </summary>        /// <param name= "msg" ></param>         Public voidSendmessagedemo (stringmsg) {            //Send String Example            varp = Producterpoolhelper.getpool (NewBusinessmqconfig () {manageconnectstring="server=192.168.17.201;initialCatalog=dyd_bs_mq_manage; User Id=sa; [Email protected]#;"},//Management Center Database                    "Dyd.mytest3");//queue path. Delimited, similar to the namespace, is the unique identifier of the queue, to inform the operation in advance to register in the message center, can be used. P.sendmessage (@"1"); //Send Object Example            /*var obj = new Message2 {text = "literal", num = 1}; var p = producterpoolhelper.getpool (new Businessmqconfig () {manageconnectstring = "server=192.168 .17.237;initial Catalog=dyd_bs_mq_manage; User Id=sa; [email protected]#;},//Management Center Database "Test.diayadian.obj");//queue path. Delimited, similar namespace, is the unique identifier of the queue, to inform operations in advance of the message            Center registration, can be used.            P.sendmessage<message> (obj); */        }        PrivateConsumerprovider Consumer; /// <summary>        ///Receiving Messages/// </summary>        /// <param name= "action" ></param>         Public voidReceivemessagedemo (action<string>action) {            if(Consumer = =NULL) {Consumer=NewConsumerprovider (); Consumer.client="Dyd.mytest3.customer1";//ClientID, the recipient of the message (consumer) unique sign, once registered, can not be changed, the business after the scrap must inform operations, delete consumer registration. Consumer.clientname ="Client Name";//This is relatively random, mainly used for their own identification, to be briefConsumer.config =NewBusinessmqconfig () {manageconnectstring ="server=192.168.17.201;initial Catalog=dyd_bs_mq_manage; UserId=sa; [Email protected]#;" };Consumer.maxreceivemqthread =1;//the number of threads in parallel processing is generally 1 enough, if the message processing is slow and you want to consume concurrently, consider the partition being used = The number of parallel processing threads is extremely optimal for parallel efficiency, but the CPU consumption should be not small. Consumer.mqpath ="Dyd.mytest3";//the queue to be received is correctConsumer.partitionindexs =Newlist<int> () {1,2,3,4,5,6,7,8};//the partition sequence number of the consumer subscription, starting with 1consumer.registerreceivemqlistener<string> (R) =                {                    /** These write business code * When writing, be careful to consider the situation of business processing failure.                       * 1. Retry failed n times. * 2. If retry is not possible, the token message has been processed.                       Then skip the message processing and document the situation yourself.                       * After the message is consumed, be sure to call markfinished and mark the message to be consumed. */action.                    Invoke (R.OBJMSG);                R.markfinished ();            }); }        }        /// <summary>        ///close a message subscription connection/// </summary>         Public voidClosereceivemessage () {//To register a consumer message, consumers must switch off (Dispose) after the program is closed. Otherwise, the exception is terminated, and the connection can be re-registered after waiting for a timeout to occur manually.             if(Consumer! =NULL) {consumer.dispose (); Consumer=NULL; }        }    } 

Part

NOTE:. NET open source Message Queuing is very small, especially for high-reliability Message Queuing for the business; Hopefully, this open source message queue will bring more solutions, more ideas and architecture to the. NET domain, and also want to know what Message Queuing people can offer more advice and advice on this solution.

Cheijiangyi

. NET Business Message Queuing