code example for writing non-blocking programs using the Python twisted framework

Take a look at the code first:

# ~*~ Twisted-a Python tale ~*~ from time import sleep # Hello, I ' m A developer and I mainly setup wordpress.def install _wordpress (Customer):  # Our hosting company Threads Ltd. is bad. I start Installation and ...  Print "Start installation for", Customer  # ... then wait till the installation finishes successfully. It is  # boring and I ' m spending the most of my time waiting while consuming  # resources (memory and some CPU cycles). It ' s because the process  # is *blocking*.  Sleep (3)  print "All-done for", Customer # I-do-all-day-long for our customersdef Developer_day (Customers):  fo R Customer in Customers:    install_wordpress (Customer) Developer_day (["Bill", "Elon", "Steve", "Mark"])

Run it and the results are as follows:

   $./deferreds.py 1

------Running Example 1------Start installation for Billall do for Billstart installation...* Elapsed time:12.03 Seco Nds

This is a sequence of code execution. Four consumers, for a person to install a time of 3 seconds, then four people is 12 seconds. This is not a very satisfying process, so take a look at the second example that uses threading:

Import Threading # The company grew. We now have many customers and I can ' t handle the# workload. We is now 5 developers doing exactly the same thing.def Developers_day (customers): # and we now has to synchronize ... a . k.a. Bureaucracy lock = Threading. Lock () # def dev_day (ID): print "goodmorning from developer", ID # yuck-i Hate locks ... lock.acquire () whi Le Customers:customer = customers.pop (0) lock.release () # My Python is less readable install_wordpres S (Customer) Lock.acquire () lock.release () print "Bye from developer", ID # We go to work in the morning devs = [Threading. Thread (Target=dev_day, args= (i,)) for I in range (5)] [Dev.start () for Dev in devs] # We leave for the evening [Dev.join () for Dev in devs] # We are get more do in the same time and our dev process got more# complex. As we grew we spend more time managing queues than doing dev# work. We even had occasional deadlocks when processes got extremely# complex. TheFact is, we are still mostly pressing buttons and# waiting and now we also spend some time in Meetings.developers_day ( ["Customer%d"% i for I in Xrange (15)])

Run it:

$./deferreds.py 2

------Running Example 2------goodmorning from developer 0Goodmorning from Developer1start installation forgoodmorning fr  Om developer 2Goodmorning from developer 3Customer 0...from Developercustomer 3Bye from developer Elapsed time:9.02 Seconds

This is a code that executes in parallel, using 5 worker threads. 15 consumers each spent 3s means a total of 45s of time, but it took 5 threads to execute in parallel for a total of only 9s of time. This code is a bit complex, and a large part of the code is used to manage concurrency rather than focusing on algorithmic or business logic. In addition, the output of the program looks very mixed, the readability of Tianjin. Even simple multithreaded code is also difficult to write well, so we switch to using twisted:

# For years we thought this is all there is ... We kept hiring more# developers, more managers and buying servers. We were trying harder# optimising processes and fire-fighting while getting mediocre# performance in return. Till Luckily one day our hosting# company decided to increase their fees and we decided to# switch to Twisted ltd.! From twisted.internet import reactorfrom twisted.internet import deferfrom twisted.internet Import Task # twisted have a SL ightly different approachdef Schedule_install (customer): # They is calling us back when a Wordpress installation complet  Es.  # They connected the caller recognition system with our CRM and # We know exactly what's a call are about and what have to be  Done next.  # # We are design processes of what have to happen on certain events. Def schedule_install_wordpress (): Def on_done (): print "callback:finished installation for", Customer Prin T "scheduling:installation for", Customer return Task.deferlater (reactor, 3, ON_done) # def All_done (_): print "All do for", "Customer # # to" customer, we schedule these processes on the CRM # and that # are all our chief-twisted developer have to do d = schedule_install_wordpress () d.addcallback (all_done # return D # Yes, we don ' t need many developers anymore or any synchronization.# ~ super-powered Twisted Developer ~ ~  def twisted_developer_day (Customers): print "goodmorning from Twisted Developer" # # Here's what is done today Work = [Schedule_install (customer) to customer in customers] # Turn off the lights when done join = defer. Deferredlist (work) Join.addcallback (lambda _: Reactor.stop ()) # print ' Bye from Twisted developer! ' # even is particularly short!twisted_developer_day (["Customer%d"% i for I in Xrange]) # Reactor, our Secreta RY uses the CRM and follows-up on Events!reactor.run ()

Operation Result:

------Running Example 3------goodmorning from Twisted developerscheduling:installation for Customer 0....scheduling:in Stallation for Customer 14Bye from Twisted developer! Callback:finished installation for customer 0All do for customer 0callback:finished installation for customer 1All Don E for customer 1...All do for customer 14* Elapsed time:3.18 seconds

This time we got the perfect execution code and readable output, and we didn't use threads. We processed 15 consumers in parallel, that is to say, 45s execution time was already completed within 3s. The trick is that we replace all of the blocking calls to sleep () with the equivalent Task.deferlater () and callback functions in twisted. With the operations now being handled elsewhere, we can effortlessly serve 15 of consumers at the same time. The
previously mentioned action occurs somewhere else. Now to explain, the arithmetic operation still occurs in the CPU, but now the CPU processing speed compared to disk and network operation is very fast. So it takes most of the time to provide data to the CPU or to send data from the CPU to memory or to another CPU. We used non-blocking operations to save this time, for example, Task.deferlater () uses a callback function that is activated when the data has been transferred to completion. Another important point of the
is the goodmorning from Twisted developer and bye from Twisted developer! information in the output. The two messages are printed when the code starts executing. If the code executes so early in this place, then when does our app actually start running? The answer is that for a twisted application (including scrapy) it runs in Reactor.run (). Before calling this method, each deferred chain that may be used in the application must be ready, and the Reactor.run () method will monitor and activate the callback function.
Note that the main rule of reactor is that you can do anything, as long as it is fast enough and is non-blocking.
Now it's okay, there's no part of the code that's used to manage multithreading, but these callback functions still seem cluttered. Can be modified to this:

# Twisted gave us utilities that make our code to more readable! @defer. Inlinecallbacksdef inline_install (Customer):  p Rint "Scheduling:installation for", Customer  yield Task.deferlater (reactor, 3, Lambda:none)  print "Callback: Finished installation for ", Customer  print" All done for ", Customer Def twisted_developer_day (Customers):  ...  Same as previously but using Inline_install () instead of Schedule_install () twisted_developer_day (["Customer%d"% i for I In Xrange ()) Reactor.run ()

The

runs the same result as the previous example. This code works the same as the previous example, but it looks more concise and straightforward. The inlinecallbacks generator can use some Python mechanisms to make the Inline_install () function pause or resume execution. The Inline_install () function becomes a deferred object and runs in parallel for each consumer. At each yield, the run is aborted on the current Inline_install () instance until the deferred object of yield is completed before resuming operation.
The only question now is, what if we have more than 15 consumers, but have, say, 10,000 consumers? This code starts at the same time 10,000 simultaneous sequences (such as HTTP requests, database writes, and so on). This may not be a problem, but it can also lead to various failures. In applications with huge concurrent requests, such as scrapy, we often need to limit the number of concurrency to an acceptable level. In the following example, we use a task. Cooperator () to complete such a function. Scrapy also uses the same mechanism in its item pipeline to limit the number of concurrent (that is, the Concurrent_items setting):

@defer. Inlinecallbacksdef Inline_install (Customer):  ... same as above # The new "problem" is that we had to manage AL L This concurrency to# avoid causing problems to others, but this is a nice problem to have.def Twisted_developer_day (Cust omers):  print "goodmorning from Twisted developer" Work  = (Inline_install (customer) for customer in customers) c3/>#  # We Use the cooperator mechanism for the Secretary not  # service more than 5 customers simultaneously.< C6/>coop = task. Cooperator ()  join = defer. Deferredlist ([Coop.coiterate (work) for I in Xrange (5)])  #  Join.addcallback (lambda _: Reactor.stop ())  Print "Bye from Twisted developer!" Twisted_developer_day (["Customer%d"% i for I in Xrange ()]) Reactor.run () # We are N  ow more lean than ever, we customers happy, our hosting# bills ridiculously low and our performance stellar.# ~*~ the END ~*~

Operation Result:

$./deferreds.py 5------Running Example 5------goodmorning from Twisted Developerbye from Twisted developer!  Scheduling:installation for customer 0...callback:finished installation for customer 4All do for customer 4Scheduling:  Installation for customer 5...callback:finished installation for customer 14All do for customer 14* Elapsed time:9.19 Seconds

As you can see from the output above, the program runs as if there were 5 slots to handle the consumer. Unless a slot is empty, it will not start processing the next consumer request. In this case, the processing time is 3 seconds, so it looks like 5 batches of processing. The final performance is the same as using threads, but this time there is only one thread, and the code is more concise and easier to write the correct code.

Ps:defertothread make synchronous functions non-blocking
Wisted's defer. Deferred (from twisted.internet import defer) can return a Deferred object.

Note: Defertothread is implemented using threads and is not recommended for excessive use
Change the synchronization function to asynchronous (return a deferred) * * *
Twisted's Defertothread (from twisted.internet.threads import Defertothread) also returns a deferred object, but the callback function is processed in another thread, primarily for the database/ File read operations

.. # code Snippet   def datareceived (self, data): Now    = Int (Time.time ())     to Ftype, data in Self.fpcodec.feed (data):      if ftype = = ' OOB ':        self.msg (' OOB: ', repr (data))      elif ftype = 0x81: # The heartbeat response to the server request (this is an analytic anti-fatigue driving device, sent to the GPS host computer, Then the host computer is sent to the server)        self.msg (' FP. PONG: ', repr (data)]      else:        self.msg (' TODO: ', (ftype, data))      D = defertothread (Self.redis.zadd, "Beier: Fpstat:fps ", now, Self.devid)      d.addcallback (Self._doresult, extra)

Here is a complete example for you to refer to

#-*-coding:utf-8-*-from twisted.internet import defer, Reactorfrom Twisted.internet.threads Import Defertothread Import functoolsimport Time # operation This is a synchronous blocking function def mysleep (timeout): TIME.SL EEP (Timeout) # The return value is equivalent to adding callback return 3 def Say (result): print "time-consuming operation is over, and give me the result of its return", "Result # with Functools.partial package Load it up, pass the parameters in. cb = Functools.partial (Mysleep, 3) d = Defertothread (CB) D.addcallback (say) print "You haven't finished, I've done it, haha" reactor.run ()