Creating a message Bus

Although the Message Queue usage is simple: pub/sub, Producer/consumer, it is really troublesome.

First, let's talk about the original requirements:

• The Web Front-end sends command messages, the backend consumer processes the messages, and the front-end obtains the results.
• Support for Windows Services

Soon, it came out:

Analysis and analysis:

• • How does the frontend know that the backend has been processed?
  • How can the front-end be triggered to execute some callback immediately after processing?
  • The Web Front-end may use ajax to regularly view the processing status of a message.

The first response is to add a response queue. At this time:

• • The frontend can be promptly notified (triggered after backend processing) to execute callback
  • However
    • How can I view Ajax-type timing information? Check in responsequeue? Obviously not (the more data in the queue, the worse the performance)
    • If the front-end is closed for a period of time, the message will be accumulated, and the performance will become worse.

Therefore, we decided to add a dB to solve the storage of these messages and subsequent Ajax-type queries, such:

Analyze and analyze the data.

• • The front-end Ajax type is not scheduled and the processing status of a message is queried multiple times.
  • If the message volume is large, you can also split rabbitmq and DB into clusters and slices respectively.
  • However, it seems that you still need to add the response queue, because after the consumer completes processing, It is very troublesome to send notifications to the front end. The reasons are as follows:
    • Inefficient notification based on DB Row Records
  • However, even if this response queue is added, the following problems may occur:
    • If the front-end crashes and the service is not on for a while, the response queue will have a backlog of messages... the performance will deteriorate.

What should I do now?

A: The pub/sub mechanism is used for this response queue. If the frontend crashes, no sub will be performed. messages will be notified only when they are online.

Therefore, continue to produce a picture

　　　　

In the figure, notifier and notifierpublisher are the front-end and back-end brokers. Considering that some threads need to actively listen, they are painted on them.

 

Let's talk about the backend. Because there are no special requirements, the requirements for the backend are as follows:

• • From the business logic perspective, messages can be processed only once without errors.
  • If an exception occurs, manual intervention is required. Messages cannot be lost, but exceptions cannot be reported in an infinite loop.
  • Manual processing is convenient for messages that report exceptions.

Separate Analysis

• • From the business logic perspective, messages can be processed only once without errors.
    • When there is no error in service processing, write back the ack action of the rabbitmq message and the message status of the DB into a transaction, as shown below:
    • This ensures that the message is "removed from the rabbitmq server and written to the query database"
    • However, if the following conditions exist, multiple executions may occur even if the business logic does not report an error.
      • That is: If no error is reported after the business logic is executed, the above Code will be triggered, but the service crash will not be triggered yet...
      • Solution
        • Add idempotence to the business logic code
        • Or
        • Add the checking nature code to the business logic code
        • Or
        • Tell me other simple methods (it is complicated to record local file logs)
  • If an exception occurs, manual intervention is required. Messages cannot be lost, but exceptions cannot be reported in an infinite loop.
    • Add the corresponding exception queue. In fact, two exception queues are added: for example:
      • For example, if messages. commanda is currently in the queue, the exception queue is:
        • Messages. commanda. exceptions1
        • Messages. commanda. exceptions2
        • Why two? Follow-up

  • Manual processing is convenient for messages that report exceptions.

    • Add a parameter to the configuration file to indicate the running level: Normal, exception 1, and exception 2.
    • If it is a common level, the consumer obtains the message from messages. commanda for processing. After an error is reported, the message is moved to prediction1.
    • If the exception level is 1, the consumer obtains the message from messages. commanda. exception1 for processing. After an error is reported, the message is moved to exception2.
    • If the exception level is 2, the consumer obtains the message from messages. commanda. exception2 for processing. After an error is reported, the message is moved to exception1.
    • There is another problem here, that is, the above rabbitmq-level messages moved from exception1 to exception2 are all completed by the action on the publish and basicack channels and cannot be set to the TX of rabbitmq, that is, consistency issues exist.
      • Similar to handlesuccessfulmessage, handlesuccessfulmessage is implemented through local dB transactions and is implemented using the base idea.

 

The other question is, does rabbitmq have the priority queue feature? The answer is:

 

Done.

 

Creating a message Bus