Use go as the right posture for RABBITMQ consumers

This is a creation in Article, where the information may have evolved or changed.

Write in front

In our production environment with two rabbitmq, the front set up a haproxy do load balancing, when our client connects to Haproxy, and then by the Haproxy is responsible for assigning the link to one of the RABBITMQ, the client needs to be responsible for wire break reconnection, It is important to have the data obtained, to assign the message to the appropriate processing method, and then to reply to the RABBITMQ ACK, where the client needs to be responsible for the logic of the wire break reconnection, because it is possible that the client and Haproxy connections are normal, But Haproxy and RABBITMQ's link because the network fluctuation is broken, then this time the client is actually not working, and will accumulate the message in the RABBITMQ unceasingly.

The following content gives a relatively perfect processing logic for reference.

Actual combat

Defining interfaces

From the previous note, this is a typical observer pattern, where the RABBITMQ object is responsible for maintaining the connection, getting the message, and then defining a number of recipients to register with the RABBITMQ object, when the Rabbitmq object receives data from RABBITMQ. The message can be distributed to the appropriate recipient to process, when the recipient processing is completed to tell the RABBITMQ object message consumption is successful, and then by the Rabbitmq object reply Rabbitmq ACK, of course, you can add a retry mechanism, the receiver may be due to a situation of processing failure, Then every certain time the RABBITMQ object needs to be re-called once the receiver is re-processed until it succeeds, and then returns an ACK.

Let's take a look at the basic interface conventions.

// Interface required by the Receiver observer mode
// Observer is used to receive data from the specified queue
type Receiver interface {
     QueueName () string // Get the queue the receiver needs to listen to
     RouterKey () string // The route bound to this queue
     OnError (error) // Handle the error encountered. When an error occurs in the RabbitMQ object, he needs to tell the receiver to handle the error
     OnReceive ([] byte) bool // Process the received message, here we need to inform the RabbitMQ object whether the message was successfully processed
}

This decouples the receiver from the RABBITMQ object so that it is easy to add a new recipient later.

Let's take a look at the definition of the RABBITMQ object:
The RABBITMQ client used here is RABBITMQ's official Github.

// RabbitMQ object used to manage and maintain rabbitmq
type RabbitMQ struct {
     wg sync.WaitGroup

     channel * amqp.Channel
     exchangeName string // name of the exchange
     exchangeType string // type of exchange
     receivers [] Receiver
}

// New creates a new RabbitMQ object
func New () * RabbitMQ {
     // Here you can define according to your needs
     return & RabbitMQ {
         exchangeName: ExchangeName,
         exchangeType: ExchangeType,
     }
}

Initialization of the RABBITMQ object

Here the Rabbitmq object needs to initialize the switch, register the receiver and initialize the queue that the receiver listens to, and the mechanism of the wire break re-connection

// prepareExchange prepare rabbitmq for Exchange
func (mq * RabbitMQ) prepareExchange () error {
    // declare Exchange
    err: = mq.channel.ExchangeDeclare (
        mq.exchangeName, // exchange
        mq.exchangeType, // type
        true, // durable
        false, // autoDelete
        false, // internal
        false, // noWait
        nil, // args
    )

    if nil! = err {
        return err
    }

    return nil
}


// run starts to get the connection and initialize the related operations
func (mq * RabbitMQ) run () {
    if! config.Global.RabbitMQ.Refresh () {
        log.Errorf ("rabbit failed to refresh connection, will reconnect:% s", config.Global.RabbitMQ.URL)
        return
    }

    // Get the new channel object
    mq.channel = config.Global.RabbitMQ.Channel ()

    // Initialize Exchange
    mq.prepareExchange ()

    for _, receiver: = range mq.receivers {
        mq.wg.Add (1)
        go mq.listen (receiver) // Each receiver starts a goroutine separately to initialize the queue and receive messages
    }

    mq.wg.Wait ()

    log.Errorf ("All tasks processing queues have exited unexpectedly")

    // In theory mq.run () will not end during the execution of the program
    // Once it ends, all the receivers have exited, which means that the program is disconnected from rabbitmq
    // then you need to reconnect, here try to destroy the current connection
    config.Global.RabbitMQ.Distory ()
}

// Start starts Rabbitmq's client
func (mq * RabbitMQ) Start () {
    for {
        mq.run ()
        
        // Once the connection is disconnected, you need to reconnect at intervals
        // It's better to have a time interval here
        time.Sleep (3 * time.Second)
    }
}

Registered Recipients

// RegisterReceiver registers a data receiver for receiving the specified route of the specified queue
func (mq * RabbitMQ) RegisterReceiver (receiver Receiver) {
    mq.receivers = append (mq.receivers, receiver)
}

// Listen to listen for messages from the specified route
// Here we need to start a goroutine for each receiver to execute listen
// This method is responsible for obtaining data from the queue that each receiver is listening on, and is responsible for retrying
func (mq * RabbitMQ) listen (receiver Receiver) {
    defer mq.wg.Done ()

    // Get the queue and route that each receiver needs to listen to
    queueName: = receiver.QueueName ()
    routerKey: = receiver.RouterKey ()

    // declare Queue
    _, err: = mq.channel.QueueDeclare (
        queueName, // name
        true, // durable
        false, // delete when usused
        false, // exclusive (exclusive queue)
        false, // no-wait
        nil, // arguments
    )
    if nil! = err {
        // When the queue initialization fails, you need to tell the receiver the corresponding error
        receiver.OnError (fmt.Errorf ("Failed to initialize queue% s:% s", queueName, err.Error ()))
    }

    // Bind Queue to Exchange
    err = mq.channel.QueueBind (
        queueName, // queue name
        routerKey, // routing key
        mq.exchangeName, // exchange
        false, // no-wait
        nil,
    )
    if nil! = err {
        receiver.OnError (fmt.Errorf ("Failed to bind queue [% s-% s] to switch:% s", queueName, routerKey, err.Error ()))
    }

    // Get the consumption channel
    mq.channel.Qos (1, 0, true) // ensure that rabbitmq will send messages one by one
    msgs, err: = mq.channel.Consume (
        queueName, // queue
        "", // consumer
        false, // auto-ack
        false, // exclusive
        false, // no-local
        false, // no-wait
        nil, // args
    )
    if nil! = err {
        receiver.OnError (fmt.Errorf ("Failed to get consumer channel for queue% s:% s", queueName, err.Error ()))
    }

    // Use callback to consume data
    for msg: = range msgs {
        // When the receiver message processing fails,
        // For example, a database connection failure caused by a network problem, redis connection failure, etc.
        // The operation can be successful by retrying, then this time you need to retry
        // do not return until the data is processed successfully, and then reply with rabbitmq ack
        for! receiver.OnReceive (msg.Body) {
            log.Warnf ("Receiver data processing failed, will try again")
            time.Sleep (1 * time.Second)
        }

        // confirm receipt of this message, multiple must be false
        msg.Ack (false)
    }
}

to integrate together

The receiver's logic is not written here, as long as the actual business logic and implementation of the interface can be, this is relatively easy.

Get the RABBITMQ connection

rabbitmqConn, err = amqp.Dial (url)
if err! = nil {
     panic ("RabbitMQ initialization failed:" + err.Error ())
}
rabbitmqChannel, err = rabbitmqConn.Channel ()
if err! = nil {
     panic ("Failed to open Channel:" + err.Error ())
}
// start and start processing data
func main () {

     // Suppose there is an AReceiver and BReceiver
     aReceiver: = NewAReceiver ()
     bReceiver: = NewBReceiver ()
    
     mq: = rabbitmq.New ()
     // Register this receiver to
     mq.RegisterReceiver (aReceiver)
     mq.RegisterReceiver (bReceiver)
     mq.Start ()
}

Application Scenarios

Give us an example of our own production environment:

We are primarily used to receive MySQL changes and incrementally update the index of the Elasticsearch, which is responsible for the database change monitoring service is Canel, it disguised as a MySQL slave, to receive the MySQL binlog change notification, The changed data is then formatted and written to RABBITMQ, which is then implemented by the go consumer to subscribe to the database for change notifications.

Since the client is not concerned about which fields in the table have changed, just know that the table specified by the database has changed, then write this change to Elasticsearch, which is the same logic for every listening table, so that we can fully configure the operation that needs to listen for table changes. I only need to specify a recipient in the config file and specify the queue to be consumed, then the program can automatically generate a number of recipients and then register into the RABBITMQ object, so we just have to write the corresponding code for some special operations, which greatly simplifies our workload, Take a look at the configuration file:

[[autoReceivers]]
    receiverName  = "article_receiver"
    database     = "blog"
    tableName    = "articles"
    primaryKey   = "articleId"
    queueName    = "articles_queue"
    routerKey    = "blog.articles.*"
    esIndex      = "articles_idx"

[[autoReceivers]]
    receiverName  = "comment_receiver"
    database     = "blog"
    tableName    = "comments"
    primaryKey   = "commentId"
    queueName    = "comments_queue"
    routerKey    = "blog.comments.*"
    esIndex      = "comments_idx"

This time you need to adjust the receiver registration function:

// WalkReceivers uses callback to iterate through all receivers
// The callback here is the one mentioned above mq.RegisterReceiver
func WalkReceivers (callback func (rabbitmq.Receiver)) {
    successCount: = 0

    // Iterate through each configuration item and generate the receivers in turn to automatically create a receiver
    // The congfig here is to obtain the configuration object uniformly. You can handle it according to the actual situation.
    for _, receiverCfg: = range config.Global.AutoReceivers {
        // verify the legitimacy of each receiver
        err: = receiverCfg.Validate ()
        if err! = nil {
            log.Criticalf ("Failed to generate% s:% s, use this configuration:% + v", receiverCfg.ReceiverName, err.Error (), receiverCfg)
            continue
        }

        // Register the receiver to the object that listens to rabbitmq
        callback (NewAutoReceiver (receiverCfg))

        log.Infof ("Generating% s successfully used this configuration:% + v", receiverCfg.ReceiverName, receiverCfg)
        successCount ++
    }

    if successCount! = len (config.Global.AutoReceivers) || successCount == 0 {
        panic ("Unable to start all receivers, please check the configuration")
    }

    // If necessary, you can continue to add recipients that need to be created manually
}

The start-up process also needs to be fine-tuned:

func registeAndStart () {
     mq: = rabbitmq.New ()

     // traverse all receivers and register them to rabbitmq
     WalkReceivers (mq.RegisterReceiver)
     log.Info ("Successfully initialized all Receivers")

     mq.Start ()
} 

This allows two receivers to be defined, and after starting the program, it is very convenient to receive changes to the database and to update the Elasticsearch index.

Written in the last

This is a summary of the usual work, hope can bring help to everyone, if there are flaws in the text, but also to correct, here the complete code is not affixed, the article has set up a complete framework, the rest is the business logic, if necessary, I will organize into a complete project on GitHub.