Getting started with the twisted framework in Python, pythontwisted

Getting started with the twisted framework in Python, pythontwisted

What is twisted?

Twisted is an event-driven network framework written in python. It supports many protocols, including UDP, TCP, TLS, and other application layer protocols, such as HTTP, SMTP, NNTM, IRC, XMPP/Jabber. A good thing is that the twisted implementation and many application-layer protocols can be used directly by developers. In fact, it is very easy to modify the implementation of the Twisted SSH server. In many cases, developers need to implement the protocol class.

A Twisted program consists of the Main Loop initiated by the reactor and some callback functions. When an event occurs, for example, when a client connects to the server, the event on the server is triggered and executed.
Use Twisted to write a simple TCP Server

The following code is a TCPServer that records the data sent from the client.

==== code1.py ====import sysfrom twisted.internet.protocol import ServerFactoryfrom twisted.protocols.basic import LineReceiverfrom twisted.python import logfrom twisted.internet import reactorclass CmdProtocol(LineReceiver):  delimiter = '\n'  def connectionMade(self):    self.client_ip = self.transport.getPeer()[1]    log.msg("Client connection from %s" % self.client_ip)    if len(self.factory.clients) >= self.factory.clients_max:      log.msg("Too many connections. bye !")      self.client_ip = None      self.transport.loseConnection()    else:      self.factory.clients.append(self.client_ip)  def connectionLost(self, reason):    log.msg('Lost client connection. Reason: %s' % reason)    if self.client_ip:      self.factory.clients.remove(self.client_ip)  def lineReceived(self, line):    log.msg('Cmd received from %s : %s' % (self.client_ip, line))class MyFactory(ServerFactory):  protocol = CmdProtocol  def __init__(self, clients_max=10):    self.clients_max = clients_max    self.clients = []log.startLogging(sys.stdout)reactor.listenTCP(9999, MyFactory(2))reactor.run()

The following code is crucial:

from twisted.internet import reactorreactor.run()

The two lines of code will start the main loop of the reator.

In the above Code, we created the "ServerFactory" class, which is responsible for returning the instance of "protocol. Each connection is processed by an instantiated "protocol" instance. Twisted's reactor will automatically create a protocol-specific instance after the TCP connection. As you can see, the methods of the protocol Class correspond to an event processing method.

When the client connects to the server, the connectionMade method is triggered. In this method, you can perform operations such as authentication, or limit the total number of client connections. Each protocol instance has a factory reference. You can use self. factory to access the factory instance.

The preceding Implementation of "Publish protocol" is a subclass of twisted. protocols. basic. LineReceiver. The LineReceiver class separates the data sent by the client according to the line break, and the lineReceived method is triggered every line break. Later, we can enhance LineReceived to parse the command.

Twisted implements its own log system. Here we configure to output logs to stdout.

When reactor. listenTCP is executed, we bind the factory to port 9999 to start listening.

user@lab:~/TMP$ python code1.py2011-08-29 13:32:32+0200 [-] Log opened.2011-08-29 13:32:32+0200 [-] __main__.MyFactory starting on 99992011-08-29 13:32:32+0200 [-] Starting factory <__main__.MyFactory instance at 0x227e3202011-08-29 13:32:35+0200 [__main__.MyFactory] Client connection from 127.0.0.12011-08-29 13:32:38+0200 [CmdProtocol,0,127.0.0.1] Cmd received from 127.0.0.1 : hello server

Use Twisted to call external processes

Next, we will add a command to the preceding server to read/var/log/syslog content.

import sysimport osfrom twisted.internet.protocol import ServerFactory, ProcessProtocolfrom twisted.protocols.basic import LineReceiverfrom twisted.python import logfrom twisted.internet import reactorclass TailProtocol(ProcessProtocol):  def __init__(self, write_callback):    self.write = write_callback  def outReceived(self, data):    self.write("Begin lastlog\n")    data = [line for line in data.split('\n') if not line.startswith('==')]    for d in data:      self.write(d + '\n')    self.write("End lastlog\n")  def processEnded(self, reason):    if reason.value.exitCode != 0:      log.msg(reason)class CmdProtocol(LineReceiver):  delimiter = '\n'  def processCmd(self, line):    if line.startswith('lastlog'):      tailProtocol = TailProtocol(self.transport.write)      reactor.spawnProcess(tailProtocol, '/usr/bin/tail', args=['/usr/bin/tail', '-10', '/var/log/syslog'])    elif line.startswith('exit'):      self.transport.loseConnection()    else:      self.transport.write('Command not found.\n')  def connectionMade(self):    self.client_ip = self.transport.getPeer()[1]    log.msg("Client connection from %s" % self.client_ip)    if len(self.factory.clients) >= self.factory.clients_max:      log.msg("Too many connections. bye !")      self.client_ip = None      self.transport.loseConnection()    else:      self.factory.clients.append(self.client_ip)  def connectionLost(self, reason):    log.msg('Lost client connection. Reason: %s' % reason)    if self.client_ip:      self.factory.clients.remove(self.client_ip)  def lineReceived(self, line):    log.msg('Cmd received from %s : %s' % (self.client_ip, line))    self.processCmd(line)class MyFactory(ServerFactory):  protocol = CmdProtocol  def __init__(self, clients_max=10):    self.clients_max = clients_max    self.clients = []log.startLogging(sys.stdout)reactor.listenTCP(9999, MyFactory(2))reactor.run()

In the above Code, the processCmd method is executed after a line of content is not received from the client. If the line of content received is the exit command, the server will disconnect the connection. If lastlog is received, we need to spit out a sub-process to execute the tail command and redirect the output of the tail command to the client. Here we need to implement the ProcessProtocol class, and we need to override the processEnded method and outpassed ED method of the class. When the tail command has output, the outReceived ED method is executed. When the process exits, the processEnded method is executed.

The following is an example of the execution result:

user@lab:~/TMP$ python code2.py2011-08-29 15:13:38+0200 [-] Log opened.2011-08-29 15:13:38+0200 [-] __main__.MyFactory starting on 99992011-08-29 15:13:38+0200 [-] Starting factory <__main__.MyFactory instance at 0x1a5a3f8>2011-08-29 15:13:47+0200 [__main__.MyFactory] Client connection from 127.0.0.12011-08-29 15:13:58+0200 [CmdProtocol,0,127.0.0.1] Cmd received from 127.0.0.1 : test2011-08-29 15:14:02+0200 [CmdProtocol,0,127.0.0.1] Cmd received from 127.0.0.1 : lastlog2011-08-29 15:14:05+0200 [CmdProtocol,0,127.0.0.1] Cmd received from 127.0.0.1 : exit2011-08-29 15:14:05+0200 [CmdProtocol,0,127.0.0.1] Lost client connection. Reason: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>: Connection was closed cleanly.

You can use the following command to initiate a command as a client:

user@lab:~$ netcat 127.0.0.1 9999testCommand not found.lastlogBegin lastlogAug 29 15:02:03 lab sSMTP[5919]: Unable to locate mailAug 29 15:02:03 lab sSMTP[5919]: Cannot open mail:25Aug 29 15:02:03 lab CRON[4945]: (CRON) error (grandchild #4947 failed with exit status 1)Aug 29 15:02:03 lab sSMTP[5922]: Unable to locate mailAug 29 15:02:03 lab sSMTP[5922]: Cannot open mail:25Aug 29 15:02:03 lab CRON[4945]: (logcheck) MAIL (mailed 1 byte of output; but got status 0x0001, #012)Aug 29 15:05:01 lab CRON[5925]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)Aug 29 15:10:01 lab CRON[5930]: (root) CMD (test -x /usr/lib/atsar/atsa1 && /usr/lib/atsar/atsa1)Aug 29 15:10:01 lab CRON[5928]: (CRON) error (grandchild #5930 failed with exit status 1)Aug 29 15:13:21 lab pulseaudio[3361]: ratelimit.c: 387 events suppressed End lastlogexit

Use a Deferred object

A reactor is a loop waiting for the occurrence of an event. The event here can be a database operation or a long computing operation. As long as these operations can return a Deferred object. The Deferred object can automatically trigger the callback function when an event occurs. The reactor blocks the execution of the current Code.

Now we need to use the Defferred object to calculate the SHA1 hash.

import sysimport osimport hashlibfrom twisted.internet.protocol import ServerFactory, ProcessProtocolfrom twisted.protocols.basic import LineReceiverfrom twisted.python import logfrom twisted.internet import reactor, threadsclass TailProtocol(ProcessProtocol):  def __init__(self, write_callback):    self.write = write_callback  def outReceived(self, data):    self.write("Begin lastlog\n")    data = [line for line in data.split('\n') if not line.startswith('==')]    for d in data:      self.write(d + '\n')    self.write("End lastlog\n")  def processEnded(self, reason):    if reason.value.exitCode != 0:      log.msg(reason)class HashCompute(object):  def __init__(self, path, write_callback):    self.path = path    self.write = write_callback  def blockingMethod(self):    os.path.isfile(self.path)    data = file(self.path).read()    # uncomment to add more delay    # import time    # time.sleep(10)    return hashlib.sha1(data).hexdigest()  def compute(self):    d = threads.deferToThread(self.blockingMethod)    d.addCallback(self.ret)    d.addErrback(self.err)  def ret(self, hdata):    self.write("File hash is : %s\n" % hdata)  def err(self, failure):    self.write("An error occured : %s\n" % failure.getErrorMessage())class CmdProtocol(LineReceiver):  delimiter = '\n'  def processCmd(self, line):    if line.startswith('lastlog'):      tailProtocol = TailProtocol(self.transport.write)      reactor.spawnProcess(tailProtocol, '/usr/bin/tail', args=['/usr/bin/tail', '-10', '/var/log/syslog'])    elif line.startswith('comphash'):      try:        useless, path = line.split(' ')      except:        self.transport.write('Please provide a path.\n')        return      hc = HashCompute(path, self.transport.write)      hc.compute()    elif line.startswith('exit'):      self.transport.loseConnection()    else:      self.transport.write('Command not found.\n')  def connectionMade(self):    self.client_ip = self.transport.getPeer()[1]    log.msg("Client connection from %s" % self.client_ip)    if len(self.factory.clients) >= self.factory.clients_max:      log.msg("Too many connections. bye !")      self.client_ip = None      self.transport.loseConnection()    else:      self.factory.clients.append(self.client_ip)  def connectionLost(self, reason):    log.msg('Lost client connection. Reason: %s' % reason)    if self.client_ip:      self.factory.clients.remove(self.client_ip)  def lineReceived(self, line):    log.msg('Cmd received from %s : %s' % (self.client_ip, line))    self.processCmd(line)class MyFactory(ServerFactory):  protocol = CmdProtocol  def __init__(self, clients_max=10):    self.clients_max = clients_max    self.clients = []log.startLogging(sys.stdout)reactor.listenTCP(9999, MyFactory(2))reactor.run()

BlockingMethod reads a file from the file system and calculates SHA1. Here we use the deferToThread method of twisted. This method returns a Deferred object. The Deferred object is returned immediately after the call, so that the main process can continue to process other events. The callback function is triggered immediately after the method passed to deferToThread is executed. If an error occurs during execution, the blockingMethod method throws an exception. If the execution is successful, the calculation result is returned through the ret of hdata.
Recommended twisted reading materials

Http://twistedmatrix.com/documents/current/core/howto/defer.html http://twistedmatrix.com/documents/current/core/howto/process.html http://twistedmatrix.com/documents/current/core/howto/servers.html

API documentation:

Http://twistedmatrix.com/documents/current/api/twisted.html