Some thoughts of message queue

In recent work, some of the solutions involved found that introducing Message Queuing would be a better and more elegant solution to the problem.

Business scenario: After the user newly renovated shop release, need related system to do some corresponding work: the cache system does the data cleanup, notifies the dependent third party system ...

Current solution: Store release system asynchronous coding to implement related logic;

Real problems: 1, the use of third-party systems to provide the interface for the store system to send notifications: Store system needs to deal with the notification of success, failure, no response logic, need to consider whether the notification must be sent to the next step; 2, without notice, directly encode the logic of the third-party system directly but the store store system: code intrusion , this logic lacks the context, it is difficult to maintain; 3. Every business party that has an interest in the store release event needs to add coding logic, and the store system is compromised by a third-party system (very big problem); 4, based on the above problems, is the third party is the use of timing updates, tolerance for a certain time period of inconsistency;

New scenario: the introduction of Message Queuing, store publishing after the release of the event, the relevant parties to the store to publish event topics, to achieve their relevant business logic;

Advantages: 1, the store system only interacts with the message queue, simplifies the store system's related coding logic, can return more quickly; 2. The Code of the third-party system is placed inside the third-party system; 3, the new business party, only need the business party to subscribe, write the relevant logic can, realize the system decoupling (very important improvement); 4 The time of system inconsistency is greatly shortened;

Some thoughts of message middleware:

Benefits: System decoupling, asynchronous, concurrency, accelerating business process processing; Through the high reliability of the message system, the high reliability of the message is ensured without adding the complicated lines of the system code: If the two systems interact directly, the sender needs the encoding processing to send the success, failure, no response and so on, in the case of the message middleware, The message's accessibility is guaranteed by the middleware in the case that the message sending middleware is successful;

Problems to be solved: reliability, no loss of message accumulation, single queue risk, message delay problem, message repetition, disorderly ordering problem, message delivery consumer mode, Message Queuing and producer, consumer cluster are associated, not to each other's cluster The impact of the expansion, the application in the system cluster subscribes to the event, causing the system to repeat the message; How the message is delivered to consumer;

Reliability: Consumer consumption capacity, downtime and so on, resulting in message accumulation, RAID magnetic consolidation column to ensure the data security of single disk, redundant disk to avoid a single point of risk, redundant queue, asynchronous replication to avoid single-queue risk, cluster to avoid the risk of individual;

Message delay: This is not a problem in many scenarios, and if so, consider whether it is possible to tolerate inconsistencies in the intermediate state within a certain period of time, if it is acceptable, and whether it can be redesigned;

Message duplication: Message delivery fails or is unresponsive, and is typically repeated, resulting in duplication. 1, the consumer power, and so on, there is no need to solve the problem of duplication; 2, the message to add a global unique ID, consumers themselves to heavy, will increase the business process response time;

Message chaos: Ordered message benefits: logic simple; problem: the maintenance of order is paid, the simplest case, only by a single producer send a success message 1 o'clock Send message 2, Message Queuing can only be sent to the same consumer sequence message 1, message 2, Greatly limits the system load capacity, even if the reference TCP after the optimization scheme: single producer parallel send with sort number of message 1, message 2, a single consumer-side combination, is still limited by the single-machine, and then through the producer and consumer content of centralized storage, Optimized for multi-producer, multi-consumer, coding difficulty and load capacity are still not ideal. So much so, in fact, is to put the orderly plan, in order to ensure the final consistency of the premise into disorder. In reality, at least 80% of the above is unordered.

Expansion: Message Queuing provides classification management for clusters and provides groupid for different consumer and producer clusters.

Cluster recurring messages: Each cluster is assigned a groupid, and each message is sent only once to one groupid;

Message Delivery Consumer: Active push, low latency, consumer active pull, frequency is difficult to control, too often waste information system resources, and occasionally high latency; consumer initiate an inquiry, if there is, send, double the number of interactions;

Some thoughts of message queue