Analysis of NSQ source code NSQD

This is a creation in Article, where the information may have evolved or changed. # # NSQ Introduction NSQ is a real-time distributed messaging platform designed to handle a large number of 1 billion-level messages per day. The NSQ has a distributed and decentralized topology that features no single point of failure, fault tolerance, high availability, and the ability to reliably deliver messages, and is a proven product that has been used in large-scale production environments. NSQ consists of 3 daemons: NSQD is the daemon that receives, saves, and transmits messages to the client. NSQLOOKUPD is the managed topology information that maintains the state of all NSQD and provides a daemon for the eventual consistent discovery service. Nsqadmin is a Web UI to monitor clusters and perform various administrative tasks in real time. [NSQ Structure Chart] (http://7xj4dv.com1.z0.glb.clouddn.com/NSQ_Topology.png) This article describes the implementation of the main NSQD. # #Topic与ChannelTopic与Channel是NSQ中重要的两个概念. The producer writes the message to topic, a topic can have multiple channel, each channel is a complete copy of the topic. Consumers subscribe to messages from the channel, and if there are multiple consumers subscribing to the same Channel,channel messages will be delivered to a random consumer.! [Image title] (http://leanote.com/api/file/getImage?fileId=564c9b31ab6441711100005b) ' type nsqd struct {//a NSQD instance can have multiple topic Topicmap map[string]*topic}type Topic struct {name string//a Topic instance has multiple Channelchannelmap map[string]* Channelmemorymsgchan Chan *message}//golang in goroutine between the communication through Chan, if you want to publish a message to the topic, just write the message to Topic.memorymsgchan Topic A new Goroutine (Messagepump) is responsible for monitoring the Topic.memorymsgchan when a new message is created, the message will be copied n sent to all channel under the topic. Func Newtopic (Topicname string) *topic {t: = &topic{name:topicname,channelmap:make (Map[string]*channel),//All Channel under the Topic Memorymsgchan:make (Chan *message, ctx.nsqd.opts.MemQueueSize), Exitchan:make (chan int),}//open a goroutine responsible for listening to write to the T OPIc Message T.waitgroup.wrap (func () {t.messagepump ()}) return T}func (t *topic) Messagepump () {var msg *messagevar Chans []*ch Annelvar Memorymsgchan Chan *message//takes out all channelfor _ under the topic, c: = Range T.channelmap {chans = append (Chans, c)}for { Remove a message from Memorymsgchan and copy the message to n copies, sent to n channel in select {Case msg = <-memorymsgchan:case <-t.exitchan:retur N}for I, Channel: = Range Chans {chanmsg: = Newmessage (Msg.id, MSG. Body) Chanmsg.timestamp = Msg. Timestamperr: = Channel. Putmessage (CHANMSG)//The message is written to the Channel.memorymsgchan of the channel}}}//channel.memorymsgchan is responsible for receiving all messages written to the channel// Creating a channel opens a new Goroutine (Messagepump) is responsible for monitoring the Channel.memorymsgchan, when there is a message will be written to the Channel.clientmsgchan, subscribe to the channel consumer will attempt to fetch messages from Clientmsgchan, a message can only be a consumer rob To//chAnnel also responsible for the reliable delivery of messages, when the message is sent to consumer, the channel will record the time the message was sent, if the time (msg-timeout parameter) did not accept consumer confirmation of the message, The channel will re-write the message to Channel.memorymsgchan and send it to the client again. Type Channel struct {name string//channel Memorymsgchan Chan *messageclientmsgchan chan *messageclients map [Int64] Consumer}func Newchannel (Topicname string, channelname string) c: = &channel{topicname:topicname,name: Channelname,memorymsgchan:make (Chan *message, ctx.nsqd.opts.MemQueueSize), Clientmsgchan:make (Chan *message), Exitchan:make (chan int),}go c.messagepump () return c}//writes a message to the channel. Func (c *channel) put (M *message) error {select {case C.memorymsgchan <-m:}return nil}func (c *channel) Messagepump () {var msg *messagefor {Select {case msg = <-c.memorymsgchan:case <-c.exitchan:goto Exit}c.clientmsgchan <-msg//Multiple A consumer will scramble for Clientmsgchan news, but only one consumer is scrambling to}exit:close (C.clientmsgchan)} ' > To understand the role of various Chan in topic channel, The key is to understand how to operate a struct in a concurrency environment (multiple Goroutine simultaneous operations topic) and lock the same structure as a golang in a C/s + + multithreaded operation(Mutex,rwmutex), the Go language generally opens a primary goroutine (Messagepump function) for the struct (Topic,channel), and all changes to the structure should be done by the master Goroutine, There is no concurrency problem, other goroutine if you want to change the structure, you should send a message (Msgchan) or notification (Exitchan,updatechan) to the fabric provided by the Chan, and the master Goroutine will always listen to all Chan , and when there is a message or notification arrives, do the appropriate processing. # # # # # # # # # # # # # # # # # # # # # Data Persistence 1. Writing a message to topic is sending the message to Topic.memorymsgchan, but Memorymsgchan is a fixed memory-sized memory queue, what if the queue is full? Will it clog? 2. If the message is stored in the Memorymsgchan memory queue, the program exits the message and all is lost? NSQ is how to solve, nsq when creating topic, channel, will create a diskqueue,diskqueue is responsible for writing messages to disk files, read messages from the disk file, is NSQ to achieve the most important structure of data persistence. In topic, for example, if a message is written to Topic.memorymsgchan but Memorymsgchan is full, NSQ writes the message to Topic.diskqueue and Diskqueue is responsible for synchronizing the message memory to disk. If the message is read from Topic.memorymsgchan, but Memorymsgchan does not have a message, the message that is synchronized to the disk file is removed from the topic.diskqueue. ' Func newtopic (topicname string,ctx *context) *topic {...//other initialization code//ctx.nsqd.opts are command-line arguments when the program starts T.backe nd = Newdiskqueue (topicname,ctx.nsqd.opts.datapath,ctx.nsqd.opts.maxbytesperfile,ctx.nsqd.opts.syncevery, Ctx.nsqd.opts.synctimeout,ctx.nsqd.opts.logger) Return t}//writes the message to the topic channel, if the ToPic Memorymsgchan will write Topic to the disk file func (t *topic) put (M *message) error {select {case T.memorymsgchan <-m:default:// A buffer interface is removed from the buffer pool, the message is written to buffer, and the buffer is written to the Topic.backend Wirtechan//buffer pool is to avoid duplicate creation destroy Buffer object B: = Bufferpoolget () T.backend.writechan <-bbufferpoolput (b)}return Nil}func (t *topic) Messagepump () {...//See code above for { Fetching messages from Memorymsgchan and Diskqueue.readchan Select {Case msg = <-memorymsgchan:case buf = <-T.backend.readchan (): msg, _ = decodemessage (buf) Case <-t.exitchan:return} ...//copy msg n copies, send to n channel under topic}} "" We see topic.ba Ckend (Diskqueue) is responsible for writing messages to disk and reading messages from disk, Diskqueue provides two Chan for external use: Readchan and Writechan. Let's take a look at some of the key points in diskqueue implementation. 1. Diskqueue opens a goroutine when it is created, reads the message from the disk file into the Readchan, the external goroutine can get the message from the Readchan, listen to Writechan at any time, When there is a message, remove the message from the Wirtechan and write to the local disk file. 2. Diskqueue to provide both the read service of the file and the write service of the file, so record the read location (readindex) and write location (writeindex) of the file. Use file every time a message is read from a file. Seek (Readindex) navigates to the file read location and then reads the message information, each time the message is written to the file. Seek (Writeindex) navigates to the write position and writes the message. 3. READINDEX,WRIteindex is important, when the program exits to write this information (meta data) to another disk file (meta-information file), the program starts by reading the meta-information file, in accordance with the meta-information file Readindex Writeindex operation to store information files. 4. Call file because the operating system layer also has a cache. Write () writes information, or it may only exist in the cache and is not synchronized to the disk, requiring the display call File.sync () to force the operating system to synchronize the cache to disk. You can control the frequency of calls to File.sync () by specifying the syncevery,synctimeout that are passed in when Diskqueue is created. Synctimeout refers to a file.sync () that is called every synctimeout second, and Syncevery is called once Syncevery () once written to the File.sync message. Both parameters can be specified by the command line when the NSQD program is started. # #网络架构nsq是一个可靠的, high-performance service-side network program, through reading NSQD source to learn how to build a reliable network server program. "//First is the listening port, when a request arrives, open a goroutine to process the link request func tcpserver (listener net. Listener) {for {clientconn, err: = Listener. Accept () Go Handle (clientconn)}}func Handle (clientconn net. Conn) {//The client first needs to send a four-byte protocol number that indicates the protocol that the client is currently using//to facilitate later smooth protocol upgrades, and the server can do different processing based on the client's protocol number BUF: = Make ([]byte, 4) _, Err: = Io. Readfull (Clientconn, buf) Protocolmagic: = String (BUF) var prot util. Protocolswitch protocolmagic {case "V2":p rot = &protocolv2{ctx:p.ctx}default:return}//Successfully established connection, process the link according to the corresponding protocol number err = Prot. Ioloop (Clientconn) return}} ' client has successfully established a link with the server, and after each client establishes a connection, NSQD creates a client interface body that holds some CLThe status information for the ient. Each client will have two goroutine, one goroutine is responsible for reading the various commands sent by clients, parsing the command, processing the command and replying the result to the client. Another goutine is responsible for sending the heartbeat information to the client regularly, and if the client subscribes to a channel, it sends the message over the network to the client in the channel. > If the server does not need to actively push a large number of messages to the client, a connection only needs to open a goroutine processing request and send a reply can be, this is the simplest way. Open two Goroutine operation the same conn, you need to pay attention to lock. "Func (P *protocolv2) ioloop (conn net. Conn) Error {//Create a new client object ClientID: = Atomic. AddInt64 (&p.ctx.nsqd.clientidsequence, 1) Client: = NewClientV2 (ClientID, Conn, p.ctx)//Turn on another goroutine, send heartbeat information regularly, The client receives the heartbeat message to reply. If NSQD has not received a heartbeat reply for the connection for a long time, it indicates that the connection has gone wrong and will disconnect, which is NSQ's heartbeat implementation Mechanism go p.messagepump (client) for {//if client is exceeded. HeartbeatInterval * 2 time interval does not receive the command sent by the client, indicating the connection problem, you need to close this link. Normally, every heartbeatinterval time the client sends a heartbeat response. Client. Setreaddeadline (time. Now (). ADD (client. HeartbeatInterval * 2))//NSQ specifies that all commands end with "\ n" and that the command and arguments are separated by a space line, err = client. Reader.readslice (' \ n ')//params[0] is the type of command, Params[1:] for the command parameter params: = bytes. Split (line, separatorbytes)//Handle client sent over command response, err: = p.exec (client, params) if err! = Nil {senderr: = p.send (client, F RametypeerroR, []byte (Err. Error ())) If _, OK: = Err. (*util. FATALCLIENTERR); OK {break}continue}//Send the processing result of the command to the client if response! = Nil {err = p.send (client, Frametyperesponse, Response)}//Connection problem , you need to close the connection conn. Close () Close (client. Exitchan)//Close the exitchan//client of the client. The channel records the channel that the client subscribes to, and the client needs to remove the subscriber from the channel when it shuts down. If client. Channel! = Nil {client. Channel.removeclient (client.id)}return Err}func (P *protocolv2) Exec (client *clientv2, params [][]byte) ([]byte, error) {switch {case bytes. Equal (Params[0], []byte ("FIN")): Return P.fin (client, params) case bytes. Equal (Params[0], []byte ("RDY")): Return P.rdy (client, params) case bytes. Equal (Params[0], []byte ("PUB")): Return p.pub (client, params) case bytes. Equal (Params[0], []byte ("NOP")): Return P.nop (client, params) case bytes. Equal (Params[0], []byte ("SUB")): Return p.sub (client, params)}return nil, util. Newfatalclienterr (Nil, "E_invalid", FMT. Sprintf ("Invalid command%s", Params[0])} "We look at some of the more important commands in NSQ:> + **nop** Heartbeat reply, no practical significance + **pub** release a message to the topic (topic) ' PUB <topic_name>\n [size of four bytes of message] [message content] ' + **sub** subscription topic (topic)/channel ' SUB &LT;TOPIC_NAME&GT ; <channel_name>\n ' + **rdy** update RDY status (indicates that the client is ready to receive n messages) ' RDY <count>\n ' + **fin** complete a message (indicates successful processing) The ' FIN <message_id>\n ' producer generates the message in a simple way, a pub command that reads a four-byte message size, then reads the message content based on the message size, and then writes the content to topic. The Messagechan. Let's focus on how the consumer reads the message from the NSQ. 1. The consumer first needs to send a sub command to tell NSQD which channel it wants to subscribe to, and then NSQD to establish a correspondence between the client and the channel. 2. The consumer sends the Rdy command, tells the server that it is ready to accept count messages, the server sends count messages to the consumer, and if the consumer wants to continue to receive the message, it needs to constantly send the Rdy command to tell the server that it is ready to accept the message (similar to the concept of sliding windows in the TCP protocol, Consumers are not in accordance with the order of consumer information, NSQD can at the same time count messages to consumers, each push to the consumer a message count number minus one, when the consumer processed the message reply fin command count+1). "Func (P *protocolv2) SUB (client *clientv2, params [][]byte) ([]byte, error) {topicname: = string (params[1]) Channeln Ame: = string (params[2]) Topic: = P.ctx.nsqd.gettopic (topicname) Channel: = topic. Getchannel (ChannelName)//Connect client to channel Channel.addclient (client.id, client) client. Channel = channel//Update message pumpclient. Subeventchan <-Channelreturn okbytes, Nil}func (P *protocolv2) messagepump (client *clientv2, Startedchan chan bool) {Subeventchan: = Clie Nt. Subeventchanheartbeatticker: = time. Newticker (client. HeartbeatInterval) for {//isreadyformessages is the Readycount that checks the client's Rdy command to determine if the message can continue to be sent to the client if subchannel = Nil | | !client. Isreadyformessages () {///client is not yet ready to set Clientmsgchan to Nilclientmsgchan = nil} else {//client ready, You try to read the message from the Clientmsgchan of the subscribed channel Clientmsgchan = subchannel.clientmsgchan}select {//receives the Rdy command sent by the client, The message is written to Readystatechan, and the following case condition is satisfied and re-entered for the For Loop case <-client. readystatechan://receives a sub command sent by the client, the message is written to Subeventchan, Subeventchan is set to nil, so a client can subscribe only once channelcase subchannel = <-subeventchan://you can ' t SUB Anymoresubeventchan = nil//send Heartbeat message case <-heartbeatchan:err = p.send (client, Frametyperesponse, Heartbeatbytes)//will have n consumers to monitor Channel.clientmsgchan, a message can only be a consumer to grab case msg, OK: = <- Clientmsgchan:if!ok {goto exit}//is sorted by the time the message is sent, the message is placed on a minimum time heap, and if a confirmation reply to the message (FIN messageId) is received within the specified time, the message is processed by the consumer and thedeleted from the heap. If the FIN messageId is not accepted for a certain amount of time, the message is removed from the heap and sent again, so NSQ can ensure that a message is consumed at least by one I. Subchannel.startinflighttimeout (msg, client.id, msgtimeout) client. Sendingmessage ()//sent to consumers via the network Err = p.sendmessage (client, MSG, &AMP;BUF) case <-client. Exitchan:goto exit}}exit:heartbeatticker.stop ()} ' # #参考文献 [NSQ Guide] (http://wiki.jikexueyuan.com/project/nsq-guide/ intro.html) [10 Reasons to use Message Queuing] (http://www.oschina.net/translate/top-10-uses-for-message-queue) [doubts about Go sync and async mode] ( Https://groups.google.com/forum/?fromgroups#!topic/golang-china/q9qClpwk5RY)