Twisted's defer mode and thread pool

Reference:http://www.cnblogs.com/mumuxinfei/p/4528910.html

Objective:
Recently helped a friend review its module service code, using Python's twisted network framework. Given that it had not been used before, it was decided to study it well.
Twisted's reactor model handles network IO events well and timed task triggers. However, the business logic operation after the packet processing needs to be decided according to the specific scenario.
This article will describe how twisted implements the Half-sync/half-async pattern, and how the thread pool and defer patterns are designed and used.

Scene Construction:
The twisted service accepts business requests and the backend needs to access MySQL. Because the MySQL interface is synchronous, its business operation (MySQL) blocks the reactor IO event loop if the installation twisted the default way to handle the session. This greatly reduces the service ability of twisted.
To solve this kind of problem, twisted supports the thread pool. Separating the business logic from the IO event, the IO operation is still asynchronous, while the business logic is handled by the thread pool.

　　

Worker thread Pool:
Before specifying the defer mode, talk about the thread pool that comes with reactor, which is also in line with the intuitive understanding of using Half-sync/half-async mode .
First, construct the next base sample code:

1234567891011121314151617181920212223242526272829303132333435

#! /usr/bin/python#-*- coding: UTF-8 -*-fromtwisted.internet importreactorfromtwisted.internet importprotocolfromtwisted.protocols.basic importLineReceiver importtimeclassDemoProtocol(LineReceiver):                   deflineReceived(self, line):        # 进行数据包的处理        reactor.callInThread(self.handle_request, line)        defhandle_request(self, line):        """            hanlde_request:                进行具体的业务逻辑处理        """        # 边使用sleep(1)来代替模拟        time.sleep(1)        # 借助callFromThread响应结果        reactor.callFromThread(self.write_response, line)        defwrite_response(self, result):        self.transport.write("ack:" +str(result) +"\r\n")classDemoProtocolFactory(protocol.Factory):    defbuildProtocol(self, addr):        return DemoProtocol()    reactor.listenTCP(9090, DemoProtocolFactory())reactor.run()

Demoprotocol receives a line of messages, it takes a second to process a business, so it calls Callinthread to execute with the reactor thread pool.
The function of its callinthread is defined as follows:

12	def  callinthread ( self * args,  * * kwargs):           self .getthreadpool (). Callinthread (_callable,  * args,   * * Kwargs)

From there, we can confirm the previous point of view and use the thread pool to complete the time-consuming and blocking business work .
Let's take a look at the function definition of callfromthread:

1234	defcallFromThread(self, f, *args, **kw):        assertcallable(f), "%s is not callable"%(f,)        self.threadCallQueue.append((f, args, kw))        self.wakeUp()

The function is to put the callback into the queue of the main thread (also reactor the main event loop) and wake up the reactor in time.
We put the write response into the main loop in order for the IO to focus on the main loop and avoid potential thread insecurity.

Defer mode:
Using reactor's thread pool directly, it is very easy to implement Half-sync/half-async mode and isolate IO and business logic. But at the beginning of reactor's design, it was more likely to hide its internal thread pool . So it introduced the defer model.
Let's implement the equivalent code snippet:

123456789101112131415161718192021222324252627282930313233343536

#! /usr/bin/python#-*- coding: UTF-8 -*-fromtwisted.internet importreactorfromtwisted.internet importprotocolfromtwisted.protocols.basic importLineReceiverfromtwisted.internet.threads importdeferToThreadimporttimeclassDemoProtocol(LineReceiver):                   deflineReceived(self, line):        # 进行数据包的处理        deferToThread(self.handle_request, line).addCallback(self.write_response)        defhandle_request(self, line):        """            hanlde_request:                进行具体的业务逻辑处理        """        # 边使用sleep(1)来代替模拟        time.sleep(1)        returnline        defwrite_response(self, result):        self.transport.write("ack:"+str(result) +"\r\n")    classDemoProtocolFactory(protocol.Factory):    defbuildProtocol(self, addr):        returnDemoProtocol()    reactor.listenTCP(9090, DemoProtocolFactory())reactor.run()

After using defer, the code is more concise. Its defer object, in fact, borrowed a thread pool.
Threads.defertothread is defined as follows:

1234567891011121314151617

defdeferToThread(f, *args, **kwargs):    fromtwisted.internet importreactor    return deferToThreadPool(reactor, reactor.getThreadPool(),                             f, *args, **kwargs)defdeferToThreadPool(reactor, threadpool, f, *args, **kwargs):    d =defer.Deferred()    defonResult(success, result):        ifsuccess:            reactor.callFromThread(d.callback, result)        else:            reactor.callFromThread(d.errback, result)    threadpool.callInThreadWithCallback(onResult, f, *args, **kwargs)     returnd

Here we can find Defertothread, is indirectly called the Callinthread function, on the other hand, the execution of its callback function results, oncallback, and Onerrback call . These callback functions run in the main thread.
The defer model simplifies program writing and also changes the way people develop their thinking patterns .

Test review:
Test with telnet and the results are normal.
　　
On the other hand, the twisted thread pool, which defaults to the way of deferred initialization .
When the service is turned on, only the main thread is one, and as the request arrives, it generates more worker threads on demand.
and its line Cheng Chime thinks 10. We can use the suggestthreadpoolsize method to modify.

Twisted's defer mode and thread pool