Golang using NSQ

This is a creation in Article, where the information may have evolved or changed.

Why to use NSQ

has recently been looking for a high-performance, highly available message queue to do communication between internal services. At first think of using ZEROMQ, but in the process of finding information, accidentally found NSQ this Golang developed message queue, after all, is Golang original things, full-featured, the key is the performance is good. Which support dynamic expansion, eliminate the single point of failure and other characteristics, can be very good to meet my needs

The previous comparison between NSQ and other MQ looks really powerful. Here is a simple record of how NSQ is used

Photo from GOLANG2017 Developer Conference

NSQ Service Side

NSQ Service-Side introduction

Before using the NSQ service, it is still necessary to understand several core components of NSQ
The entire NSQ service consists of three main parts

Nsqlookupd

Let's take a look at what the official words say:
NSQLOOKUPD is the daemon responsible for managing topology information. The client discovers the producer of the specified topic (topic) by querying NSQLOOKUPD, and the NSQD node broadcasts the topic (topic) and channel information

Simply put, NSQLOOKUPD is the central management Service, which uses TCP (default port 4160) to manage the NSQD service, using HTTP (the default port 4161) to manage the Nsqadmin service. Also provides query functionality for clients

In general, NSQLOOKUPD has the following features or features

• Uniqueness, there is only one NSQLOOKUPD service in a NSQ service. It is also possible to deploy multiple nsqlookupd in the cluster, but there is no association between them

• De-NSQLOOKUPD, even if it crashes, will not affect the running NSQD service

• Serves as a middleware for NSQD and naqadmin information interactions

• Provides an HTTP query service to periodically update the NSQD address directory for clients

Nsqadmin

Official quote: A WEB UI that aggregates real-time statistics for a cluster and performs different management tasks

In general, Nsqadmin has the following features or features

• Provide a unified management of topic and channel operation interface and a variety of real-time monitoring data display, interface design is very simple, easy to operate

• Show the number of all the message, uh .... Pack x Sharp

• Be able to create topic and channel in the background, this should not be used to

• All features of Nsqadmin must rely on nsqlookupd,nsqadmin to simply pass user actions to NSQLOOKUPD and show data from NSQLOOKUPD

Nsqadmin The default access address is http://127.0.0.1:4171/

Nsqd

Official quote: NSQD is a daemon that is responsible for receiving, queuing, and delivering messages to clients

Simply put, the real work is the service, it is mainly responsible for the message of the transceiver, queue maintenance. NSQD will listen to a TCP port (4150) and an HTTP port (4151) and an optional HTTPS port by default

In general, NSQD has the following features or features

• For subscribers to the same topic, the same channel consumer uses a load balancing policy (not polling)

• As long as the channel exists, even consumers without the channel will cache the producer's message in the queue (note the expiration of the message)

• The message in the queue is guaranteed to be consumed at least once, even if NSQD exits, the messages in the queue are staged on the disk (except for unexpected cases such as ending the process)

• Limited memory consumption, ability to configure the number of message caches in memory for each channel queue in NSQD, and once exceeded, message will be cached to disk

• Topic,channel once established, will always exist, in a timely manner in the management station or in code to remove invalid topic and channel, to avoid waste of resources

This is the official figure, the first channel (meteics) because there are multiple consumers, so triggered the load balancing mechanism. After two channel because there is no consumer, all the message will be cached in the corresponding queue until the consumer appears

One problem here is that if a channel is only being delivered by a producer, will it cause server resources to be exhausted? Perhaps the NSQD internally did the corresponding treatment, but to avoid the occurrence of this situation

NSQ server-to-client relationship

Understand the relationship between NSQLOOKUPD,NSQD and consumers and producers in the client

Consumers

Consumers have two ways of connecting with NSQD

• Consumers directly connected to the NSQD, this is the simplest way, the disadvantage is that the NSQD service can not achieve dynamic scaling (of course, to achieve one is also possible)

• The consumer obtains the connection address of all NSQD on the NSQLOOKUPD via HTTP query nsqlookupd and then establishes a connection with the NSQD (the official recommended practice), but the client constantly queries the NSQLOOKUPD for the latest NSQD address directory ( Don't like polling this way with http ...)
  Or take a closer look at the chart, the official consumer model:

Producers

Producers must directly connect NSQD to deliver a message (online said, can connect to NSQLOOKUPD, let nsqlookupd automatically choose a nsqd to complete delivery, but I use producer TCP is not even nsqlookupd, I don't know if HTTP can ...),

One problem here is that if the producer is connected to the NSQD, then the message will fail, so the client must implement the corresponding fallback plan.

Installing NSQ

Method One

• First build the Golang development environment, here is not detailed

  • Note that it is best to add the bin directory to the System environment (PATH) when building the Golang environment, eliminating the hassle of going to the bin directory every time

• Install Package Manager GODEP

go get github.com/tools/godep

After the execution, check whether the GODEP is installed in the bin directory, the general will automatically install, if not, with Go install manual installation under

• Install Dependency Pack assert

go get github.com/bmizerany/assert

• Installing NSQ

godep get github.com/bitly/nsq/...

If the installation succeeds, a lot of nsq_ will appear in the Bin directory ... The beginning of the executable file

• PS: If the installation fails

  • Like me there's a whole bunch of "use of the internal package not allowed" error, looking for half a day to find this sentence in a corner
    Note: NSQ maintains go get compatibility, but is not recommended because it is not guaranteed to still compile.

  At this time the method of two installation

Method Two

• Go directly to https://github.com/nsqio/nsq/releases download the compiled execution file, copy the executable file inside the bin directory to use the

Run NSQ

Running a standalone NSQD service

NSQD is an independent service, start a NSQD can complete message transceiver, start a single-machine nsqd, very simple

nsqd

The client can use HTTP, or TCP, where I use the official GO-NSQ package as the client, using TCP to send and receive message

• Send
  

• Receive
  

Running the NSQ service cluster

• Start Nsqlookud First

nsqlookupd

• Start Nsqd and access the Nsqlookud that just started. Here, to facilitate the next test, two Nsqd are launched.

nsqd --lookupd-tcp-address=127.0.0.1:4160

nsqd --lookupd-tcp-address=127.0.0.1:4160 -tcp-address=0.0.0.0:4152 -http-address=0.0.0.0:4153

• Start Nqsadmin

nsqd --lookupd-http-address=127.0.0.1:4161

GO-NSQ-based client implementation

A few notable places

• Producer will not be re-connected after disconnection, you need to manually re-connect, consumer disconnection will automatically re-connect

• Consumer's re-connect time configuration item has two functions (this design must be spit out, a bit better separately)

  • Consumer detects that a connection to the NSQD has been broken, and every x seconds to the NSQD request to reconnect

  • Consumer every x seconds, HTTP polling to Nsqlookud to update their NSQD address directory

  • The consumer time of the default is 60s (...). The dishes are cold), I changed to 1s

• Consumer can receive the same name topic data from different NSQD node at the same time, in order to avoid confusion, it must be processed on the client

• The interface callbacks set in Addconurrenthandlers and AddHandler are performed in a different goroutine

• Producer cannot publish (Publish) empty message, otherwise it will cause panic

Go_nsq-send.go

 //nsq send the Test package mainimport ("Bufio" "FMT" "GITHUB.COM/NSQIO/GO-NSQ" "Os") var producer *nsq. producer//Main function func main () {strIP1: = "127.0.0.1:4150" strIP2: = "127.0.0.1:4152" Initproducer (strIP1) Running: = TRUE//Read Take console input Reader: = Bufio. Newreader (OS. Stdin) for running {data, _, _: = Reader. ReadLine () Command: = string (data) if command = = "Stop" {running = false} for err: = Publish ("Test", command); Err! = Nil; Err = Publish ("Test", command) {//Toggle IP re-connect strIP1, strIP2 = strIP2, strIP1 initproducer (strIP1)}}}//Initialize producer Func Initprodu CER (str string) {var err error FMT. PRINTLN ("Address:", str) producer, err = nsq. Newproducer (str, NSQ. Newconfig ()) if err! = Nil {panic (err)}}//Post message func Publish (topic string, message String) error {var err error if Produc Er! = Nil {if message = = "" {//cannot publish empty string, otherwise it will cause error return nil} err = producer. Publish (topic, []byte (message))//Publish message Return ERR} return FMT. Errorf ("Producer is nil", err)}  

Go_nsq-receive.go

NSQ Receive Test Package Mainimport ("FMT" "Time" "github.com/nsqio/go-nsq")//consumer type Consumert struct{}//main function func main () {Init Consumer ("Test", "Test-channel", "127.0.0.1:4161") for {time. Sleep (time. Second * 10)}}//handles message func (*consumert) handlemessage (msg *nsq. Message) Error {FMT. PRINTLN ("Receive", MSG. Nsqdaddress, "Message:", String (Msg. Body)) return nil}//Initialize consumer func initconsumer (topic string, channel string, address string) {cfg: = nsq. Newconfig () cfg. Lookupdpollinterval = time. Second          //Set the re-connect time c, err: = nsq. Newconsumer (topic, Channel, CFG)//New consumer if err! = Nil {panic (err)} c.setlogger (nil, 0)        // Shield System Log C.addhandler (&consumert{})//Add consumer interface//Establish NSQLOOKUPD connection If err: = C.CONNECTTONSQLOOKUPD (address); Err! = Nil {panic (ERR)}//Establish multiple NSQD connections//If ERR: = C.connecttonsqds ([]string{"127.0.0.1:4150", "127.0.0.1:4152"}); Err! = Nil {//  Panic (ERR)//}//Establish a NSQD connection//If Err: = C.connecttonsqd ("127.0.0.1:4150"); Err! =Nil {//  Panic (ERR)//}}