The Python Message Queuing package SNAKEMQ uses

Using Message Queuing has many advantages in data communication, SNAKEMQ is an open-source, Python-implemented cross-platform MQ Library, Well,python's Message Queuing package SNAKEMQ Use, here we go:

I. Official introduction of SNAKEMQ
SNAKEMQ GitHub project page: https://github.com/dsiroky/snakemq1. Pure python implementation, cross-platform

2. Automatic Reconnection

3. Reliable send-configurable message mode and message time-out mode

4. Persistent/temporary two types of queues

5. Support Asynchronous--poll ()

6.symmetrical--A single TCP connection can be used for duplex communication

7. Multi-database Support--SQLite, MongoDB ...

8.brokerless-Similar to ZEROMQ implementation principle

9. Expansion module: RPC, Bandwidth throttling

All of the above are Mandarin, need to verify their own, hands-on encapsulation, feel Meng Meng da.

Ii. description of several major issues

1. Support automatic re-connect, do not need to write their own heartbeat logic, you just need to focus on sending and receiving on the line

2. Support data persistence, if you start to persist, the data will be sent automatically after you re-connect.

3. Data reception, SNAKEMQ by providing a callback implementation, you only need to write a receive method to add to the callback list.

4. The sending of data, which is sent here is the bytes type (binary), and therefore needs to be converted. I test in the program is a text string, using Str.encode (' Utf-8 ') to convert to bytes, received and then converted back.

5. Terminology interpretation, Connector: Similar to the socket of the Tcpclient,lisenter: similar to the socket tcpserver, each Connector or listener is a ident identity, When you send and receive data, you know whose data it is.

6. When using SQLite persistence, you need to modify the source code, Sqlite3.connect (Filename,check_same_thread = False), to solve the problem of multi-threaded access to SQLite. (Will it deadlock?) ）

7. When the persistence is started, if it is re-connected, it will be sent automatically to ensure reliable.

8. In order to encapsulate the need, after the data receive, I pass the callback way to send out.

Third, the Code

A custom log module is used in the description code

From common import Nxloggerimport Snakemqlogger as Logger

Can be replaced by logging.

Callback Class (callbacks.py):

#-*-Coding:utf-8-*-"synchronized Callback" class callback (Object):  def __init__ (self):    self.callbacks = []   def add (Self, func):    self.callbacks.append (func)   def remove (self, func):    Self.callbacks.remove ( Func)   def __call__ (self, *args, **kwargs): For    callback in Self.callbacks:      callback (*args, **kwargs)

Connector Class (snakemqconnector.py):

#-*-Coding:utf-8-*-import threadingimport snakemqimport snakemq.linkimport snakemq.packeterimport Snakemq.messagingimport snakemq.messagefrom snakemq.storage.sqlite Import Sqlitequeuesstoragefrom snakemq.message Import flag_persistentfrom common.callbacks import Callback from common import nxloggerimport Snakemqlogger as Logger clas S Snakemqconnector (threading.         Thread): Def __init__ (self, snakemqident = None, Remoteip = "localhost", remoteport = 9090, persistent = False): Super (Snakemqconnector,self). __init__ () self.messaging = None Self.link = None Self.snakemqiden t = snakemqident Self.pktr = None Self.remoteip = Remoteip Self.remoteport = RemotePort SE Lf.persistent = Persistent self.on_recv = Callback () self._initconnector () def run (self): Log Ger.info ("connector start ...") if self.link! = None:self.link.loop () logger.info ("      Connector end ... ")   def terminate (self): Logger.info ("connetor terminating ...") if self.link! = None:self.lin K.stop () Self.link.cleanup () logger.info ("Connetor terminated") def on_recv_message (self, conn,         Ident, message): Try:self.on_recv (Ident, Message.data.decode (' Utf-8 ')) #dispatch received data Except Exception as E:logger.error ("Connector recv:{0}". Format (e)) print (e) "Send mess Age to dest host named Destident "Def sendmsg (self, destident, byteseq): msg = None If self.persiste Nt:msg = Snakemq.message.Message (Byteseq, ttl=60, flags=flag_persistent) else:msg = sn Akemq.message.Message (BYTESEQ, ttl=60) if self.messaging = = None:logger.error ("Connector:messaging I         s not initialized, send message failed ") return Self.messaging.send_message (destident, msg) ' "Def _iniTconnector (self): Try:self.link = Snakemq.link.Link () self.link.add_connector ((Self.rem                  OTEIP, self.remoteport)) Self.pktr = Snakemq.packeter.Packeter (self.link) if self.persistent: Storage = Sqlitequeuesstorage ("snakemqstorage.db") self.messaging = Snakemq.messaging.Me Ssaging (Self.snakemqident, "", SELF.PKTR, storage) else:self.messaging = snakemq.messaging. Messaging (Self.snakemqident, "", Self.pktr) Self.messaging.on_message_recv.add (self.on_recv_mess              Age) Except Exception as E:logger.error ("connector:{0}". Format (e)) finally: Logger.info ("connector[{0}] Loop ended ...". Format (self.snakemqident))

Listener Class (snakemqlistener.py):

#-*-Coding:utf-8-*-import threadingimport snakemqimport snakemq.linkimport snakemq.packeterimport Snakemq.messagingimport snakemq.messagefrom Common import nxloggerimport Snakemqlogger as Loggerfrom common.callbacks Import Callbackclass Snakemqlistener (threading. Thread): Def __init__ (self, snakemqident = None, ip = "localhost", port = 9090, persistent = False): Super (Sna kemqlistener,self). __init__ () self.messaging = None Self.link = None Self.pktr = None Self         . snakemqident = snakemqident Self.ip = IP;         Self.port = Port self.connectors = {} SELF.ON_RECV = Callback () self.persistent = persistent                  Self._initlistener () "Thread Run" def run (self): Logger.info ("Listener start ...") If Self.link! = None:self.link.loop () logger.info ("Listener end ...") "Termin     ATE SNAKEMQ listener thread "def terminate (self):    Logger.info ("Listener terminating ...") if self.link! = None:self.link.stop () self.l Ink.cleanup () logger.info ("Listener terminated") ' Receive message from host named Ident ' De F On_recv_message (SELF, conn, ident, message): Try:self.on_recv (Ident, Message.data.decode (' Utf-8 ')) #dispatch received Data self.sendmsg (' Bob ', ' hello,{0} '. Format (ident). Encode (' Utf-8 ')) except Exceptio N as E:logger.error ("Listener recv:{0}". Format (e)) print (e) def on_drop_message (self, Iden T, message): Print ("message dropped", ident, message) logger.debug ("Listener:message dropped,ident:{0},mes Sage:{1} ". Format (ident, message)) ' Client Connect ' def on_connect (self, ident): Logger.debug (" Listene R:{0} Connected ". Format (ident)) self.connectors[ident] = ident self.sendmsg (ident," Hello ". Encode (' Utf-8 ') ) "Client DisconnECT ' Def on_disconnect (self, ident): Logger.debug ("listener:{0} disconnected". Format (ident)) if Iden T in Self.connectors:self.connectors.pop (ident) "Listen start loop" Def _initlistener (self): Try:self.link = Snakemq.link.Link () Self.link.add_listener ((Self.ip, Self.port)              ) Self.pktr = Snakemq.packeter.Packeter (self.link) self.pktr.on_connect.add (self.on_connect) Self.pktr.on_disconnect.add (self.on_disconnect) If self.persistent:storage = Sq Litequeuesstorage ("snakemqstorage.db") self.messaging = snakemq.messaging.Messaging (Self.snakemqident, "" , SELF.PKTR, storage) else:self.messaging = snakemq.messaging.Messaging (Self.snakemqident, "", Self.pktr) Self.messaging.on_message_recv.add (self.on_recv_message) Self.messag Ing.on_message_drop.add (Self.on_drop_message) except Exception as E:logger.error ("listener:{0}". Format (e)) finally: Logger.info ("Listener:loop ended ...") "Send message to dest host named Destident" Def sendmsg (SE LF, Destident, byteseq): msg = None If self.persistent:msg = Snakemq.message.Message (byteseq , ttl=60, flags=flag_persistent) else:msg = Snakemq.message.Message (Byteseq, ttl=60) if self . Messaging = = None:logger.error ("listener:messaging is not initialized, send message failed") r Eturn self.messaging.send_message (destident, msg)

Test Code Connector (testsnakeconnector.py):

Read the local 1M file and send it to listener, and then listener send back a hello message.

From NETCOMM.SNAKEMQ import Snakemqconnectorimport timeimport sysimport Osdef received (ident, data): print (data) if __ name__ = = "__main__": Bob = Snakemqconnector.snakemqconnector (' Bob ', "10.16.5.45", 4002,true) Bob.on_recv.add (receiv              ed) Bob.start () Try:with open ("testfile.txt", encoding= ' Utf-8 ') as F:txt = F.read () For I in range: bob.sendmsg ("Niess", Txt.encode (' Utf-8 ')) Time.sleep (0.1) E Xcept Exception as E:print (e) time.sleep (5) bob.terminate () test Code Listener (testsnakelistener.py): from NE TCOMM.SNAKEMQ Import Snakemqlistenerimport Time Def received (ident, data): filename = "log/recfile{0}.txt". Format (Time . Strftime ('%s ', Time.localtime ())) file = open (filename, ' W ') file.writelines (data) file.close () if __name__ = = "__main__": niess = Snakemqlistener.snakemqlistener ("niess", "10.16.5.45", 4002) Niess.on_recv.add (Received) NIE Ss.start () print ("NiESS start ... ") Time.sleep (Niess.terminate () print (" Niess end ... ")