Twisted server development skills (2)

Source: Internet
Author: User

The second optimization method can be used only for explanation. That is, all requests are preprocessed using a lightweight thread pool (PreProcess), and all requests that do not require short I/O execution time are directly executed, if disk I/O is required, it is placed in the next-level blocking queue, and there is a separate thread pool to process these requests. For details, see:

 


The first-level request uses its own existing thread pool. The I/O Request + second-level thread pool can be implemented using the ThreadPool mechanism provided by twisted. The optimization I mentioned uses this method, and the code is very simple, as follows:

Deferred = threads. deferToThread (data_loader.get, sn)

Deferred. addCallback (self. loader_callback, (req, other_data ))

Explanations:

Threads. deferToThread will put data_loader.get into the queue of the reactor thread pool and return a defer object. Data_loader.get is executed by the reactor thread pool and put into the reactor queue after execution is complete. Then, the reactor main thread calls the callback function registered in deferred. addCallback. Therefore, callback functions are not called across threads. If you call some applications that cannot be called across threads (such as the memcached client) in the callback function, you can safely use them, this is also one of the reasons for choosing the reactor thread pool as the second-level thread pool.

Select the reactor thread pool as the second-level thread pool. Reason 2: callback function. Because Read threads are responsible for recovering requests, callback functions are essential.

Next we will go deep into the twisted source code to explore the principles of this method. The following code is excerpted from twisted2.0.0 source code. Other versions are roughly the same:

[Python]
Def deferToThread (f, * args, ** kwargs ):
D = defer. Deferred ()
From twisted. internet import reactor
Reactor. callInThread (_ putResultInDeferred, d, f, args, kwargs)
Return d
Def _ putResultInDeferred (deferred, f, args, kwargs ):
From twisted. internet import reactor
Try:
Result = f (* args, ** kwargs)
Except t:
F = failure. Failure ()
Reactor. callFromThread (deferred. errback, f)
Else:
Reactor. callFromThread (deferred. callback, result)
----- From threads. py

[Python] view plaincopy
Def callInThread (self, _ callable, * args, ** kwargs ):
If not self. threadpool:
Self. _ initThreadPool ()
Self. threadpool. callInThread (_ callable, * args, ** kwargs) // The read operation is performed by the thread pool.
Def callFromThread (self, f, * args, ** kw ):
...
Self. threadCallQueue. append (f, args, kw) // put it into the main thread queue and the main thread executes the callback function.
Self. wakeUp ()
...
----- From base. py

Note: callInThread/allFromThread is put into the thread pool for execution, and the latter is in the reactor queue for execution by the main thread of the reactor.

The threadpool code is a thread pool in twisted/python/threadpool.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.