Analyze the core features of Python's Twisted framework

This article mainly introduces the core features of Python's Twisted framework, including key usage of reacter and Deferred in the Twisted framework. For more information, see the next. reactor
The core of twisted is reactor. what reactor mentioned is inevitable is synchronous/asynchronous, blocking/non-blocking. in Dave's first conceptual introduction, the limitations of synchronous/asynchronous are a bit vague, for more information about synchronous/asynchronous, blocking, and non-blocking, see. The proactor and reactor are described in detail in this recommendation blog.
For network IO in reactor mode, it should be synchronous IO instead of asynchronous IO. The core of the asynchronization mentioned in Dave's first chapter is: explicitly giving up control of the task rather than being randomly stopped by the operating system, programmers must organize tasks into sequences for small steps to complete. Therefore, if one of the tasks uses the output of another task, the dependent tasks (that is, the tasks that receive the output) need to be designed to receive the series of bits or fragments instead of all received.
Explicitly giving up the control of a task is a bit similar to the way you think about the coroutine. The reactor can look at the scheduler of the coroutine. Reactor is an event loop. we can register events of interest (such as socket-readable/writable) and processors (such as read/write operations) with the reactor ), the reactor calls back our processor when an event occurs. after the processor completes execution, it is equivalent to a coroutine suspension (yield) and returns to the reactor event loop, waiting for the next event to come and callback. The reactor itself has a Synchronous Event Demultiplexer, which can be implemented by the select/epoll mechanism. of course, the twisted reactor Event trigger is not necessarily based on IO, it can also be triggered by timer and other mechanisms.
The reactor of twisted does not need to register events and callback functions. Instead, it is implemented through polymorphism (inheriting specific classes, implementing the event interfaces of interest, and then passing them to twisted reactor. There are several notes about twisted reactor:
Twisted. internet. reactor is a singleton mode. each program can have only one reactor;
The reactor callback function should be used to complete operations as soon as possible. do not execute blocking tasks. the reactor is essentially a single thread. the user callback code runs in the same context as the twisted code, and a callback function is blocked, this will cause loop blocking of the entire reactor event;
Reactor will always run, unless it is stopped through reactor. stop (), but generally calling reactor. stop () means that the application ends;

II. simple use of twisted
The essence of twisted is reactor. we can use the underlying API of twisted (without the convenient high-level abstraction of twisted) to use reactor:

# Example 1 use the underlying twisted API from twisted. internet import reactofrom twisted. internet import mainfrom twisted. internet. interfaces import IReadDescriptorimport socket class MySocket (IReadDescriptor): def _ init _ (self, address): # connect to the server self. address = address self. sock = socket. socket (socket. AF_INET, socket. SOCK_STREAM) self. sock. connect (address) self. sock. setblocking (0) # tell the Twisted reactor to monitor this socket for reading reactor. addReader (self) # interface: Tell the reactor to listen to the socket descriptor def fileno (self): try: return self. sock. fileno () blocks T socket. error: return-1 # interface: def connectionLost (self, reason): self. sock. close () reactor. removeReader (self) # called when the application needs to be terminated: # reactor. stop () # interface: When the socket descriptor has readable data, def doRead (self): bytes = ''# read as much data as possible while True: try: bytesread = self. sock. recv (1024) if not bytesread: break else: bytes + = bytesread failed t socket. error, e: if e. args [0] = errno. EWOULDBLOCK: break return main. CONNECTION_LOST if not bytes: return main. CONNECTION_DONE else: # parse the protocol and process data print bytes

Example 1 clearly shows the nature of twisted reactor: add a listener descriptor and listen for readable/writable events. When an event comes with a temporary callback function, the listener continues to listen for the event after the callback is complete.
Note:
The socket is not blocked. if it is blocked, the meaning of the reactor is lost.
We provide the interfaces required by reactor by inheriting IReadDescriptor.
Add the socket class to the reactor listening object through reactor. addReader
Main. CONNECTION_LOST is the predefined value of twisted. with these values, we can control the next callback to a certain extent (similar to simulating an event)
However, the above MySocket class is not good enough and has the following disadvantages:
We need to read data by ourselves, rather than the framework to help us read and handle exceptions.
Network I/O and data processing are mixed and not separated.

III. twisted abstraction
Based on the reactor, twisted establishes a Higher Abstraction. for a network connection, twisted establishes the following three concepts:
Transports: network connection layer. it is only responsible for network connection and reading/writing bytes of data.
Protocols: the protocol layer, service-related network protocol, which replaces byte transfer with the data required by the application
Protocol Factories: a Protocol factory that creates Protocols. each network connection has a Protocols object (because the Protocol resolution status needs to be saved)
These concepts of twisted are similar to the ranch network framework in erlang. The ranch framework also abstracts the Transports and Protocols concepts. when there is a new network connection, ranch automatically creates Transports and Protocols, protocols is input when you start ranch. it is a module that implements ranch_protocol behaviour. when Protocols is initialized, it receives the Transports corresponding to the connection, in this way, we can process byte stream data in Protocols, parse and process data according to our protocol. At the same time, you can use Transports to send data (ranch has helped you read byte stream data ).
Similar to ranch, twisted will also create Protocols and pass in Transport when a new connection arrives. twisted will help us read byte stream data. we only need to read the data in dataReceived (self, data) to process byte stream data. At this time, twisted can be truly asynchronous on network I/O. It helps us to handle network I/O and possible exceptions, and disconnects network I/O and data processing, abstract as Transports and Protocols, improving program clarity and robustness.

# Example 2 use twisted abstract from twisted. internet import reactorfrom twisted. internet. protocol import Protocol, ClientFactoryclass MyProtocol (Protocol): # interface: called during Protocols initialization, and pass in Transports # In addition, twisted will automatically set the Protocols factory object member as a reference of the ProtocolsFactory instance # so that the factory can interact with MyProtocolFactory def makeConnection (self, trans ): print 'make connection: get transport: ', trans print 'My factory is:', self. factory # interface: def dataReceived (self, data): self. poem + = data msg = 'task % d: got % d bytes of poetry from % s' print msg % (self. task_num, len (data), self. transport. getPeer () # interface: disconnect def connectionLost (self, reason): # process of disconnections class MyProtocolFactory (ClientFactory): # interface: indicates the Protocols protocol = PoetryProtocol # tell base class what proto to build def _ init _ (self, address): self. poetry_count = poetry_count self. poems ={}# task num-> poem # interface: call back def buildProtocol (self, address): proto = ClientFactory when Protocols is created. buildProtocol (self, address) # Initialize proto .... return proto # interface: def clientConnectionFailed (self, connector, reason): print 'failed' to connect to: ', connector. getDestination () def main (address): factory = MyClientFactory (address) host, port = address # Input ProtocolsFactory reactor when connecting to the server. connectTCP (host, port, factory) reactor. run ()

Example 2 is much simpler and clearer than example 1 because it does not need to process network I/O and is more logically clear, in fact, ClientFactory and Protocol provide more interfaces for more flexible and powerful logic control. for specific interfaces, see twisted source code.

4. twisted Deferred
The twisted Deferred object is used to solve this problem: sometimes we need to embed our own callback in ProtocolsFactory so that an event occurs in Protocols (for example, all Protocols are processed, calls back the specified function (such as TaskFinished ). If we implement callback by ourselves, we need to solve several problems:
How do I distinguish between correct and error responses for Callback? (When using asynchronous calls, pay special attention to the importance of error return)
What if we need to execute a public function (for example, close the connection) for both correct and error responses?
If this callback is only called once?
The Deferred object is used to solve this problem. it provides two callback chains, corresponding to the correct return and error return. when the correct return or error return are returned, it calls the functions in the corresponding chain in sequence, and ensure the uniqueness of the callback.

D = Deferred () # add correct callback and error callback d. addCallbacks (your_ OK _callback, your_err_callback) # Add a public callback function d. addBoth (your_common_callback) # The correct return will call your_ OK _callback (Res)-> common_callback (Res) d. callback (Res) # The returned error will call your_err_callback (Err)-> common_callback (Err) d. errback (Err) # Note: For the same Defered object, only one request can be returned. if you try to return multiple requests, an error is returned.

The defer of twisted is an asynchronous monetization method. The difference between him and thread is that he is based on time events.
With deferred, you can manage and control the execution of tasks. To prevent programs from running, the program is stuck in blocking due to waiting for the completion of a task, improving the overall running efficiency.
Deferred can help you write asynchronous code, but it does not automatically generate asynchronous or non-blocking code! To program a synchronous function as an asynchronous function, you must return Deferred in the function and register the callback correctly.

5. Comprehensive examples

In the following example, you can run it on your own. I have mentioned some scattered examples above. let's go through the complete examples below. Twisted is actually a little difficult to understand. as long as you know that it is based on events, you can understand it slowly.

# Coding: UTF-8 # xiaorui. ccfrom twisted. internet import reactor, deferfrom twisted. internet. threads import deferToThreadimport OS, sysfrom twisted. python import threadable; threadable. init (1) deferred = deferToThread. _ get _ import timedef todoprint _ (result): print resultdef running (): "Prints a few dots on stdout while the reactor is running. "# sys. stdout. write (". "); sys. stdout. flush () print '. 'reactor. CallLater (. 1, running) @ deferreddef sleep (sec): "A blocking function magically converted in a non-blocking one. "print 'start sleep % s' % sec time. sleep (sec) print '\ nend sleep % s' % sec return "OK" def test (n, m): print "fun test () is start "m = m vals = [] keys = [] for I in xrange (m): vals. append (I) keys. append ('a % s' % I) d = None for I in xrange (n): d = dict (zip (keys, vals) print "fun test () is end "ret Urn dif _ name __= = "_ main _": # one sleep (10 ). addBoth (todoprint _) reactor. callLater (. 1, running) reactor. callLater (3, reactor. stop) print "go !!! "Reactor. run () # two aa = time. time () de = defer. deferred () de. addCallback (test) reactor. callInThread (de. callback, 10000000,100) print time. time ()-aa print "I will do something else here" print de print "go end"

For more information about the core features of the Python Twisted framework, see The PHP Chinese web!