Python Twisted Learning Series 19 (reprinted stulife The best Twisted Introductory tutorial)

Part 19th change the idea before

Introduction Twisted is a project that is progressing, and its developers regularly add new features and extend the old features. With Twisted 10.1.0 released, developersDeferredClass adds a new feature--Cancellation-That's what we're going to look at today.

Asynchronous programming decouples the request from the response, thus bringing a new possibility: between the request result and the return result, you may decide that you no longer need the result. Consider:d OC: The poetry proxy server in ' P14 '. Here's how this works, at least for the first time in poetry:

1. A request for poetry came.
2. This agent is in practice with the server to get this poem
3. Once the poem is complete, send it to the original requesting agent

It looks perfect, but what if the client hangs up before it gets a poem? Perhaps they had previously requested the entire contents of the Paradise Lost, and then they decided that what was actually wanted was Kojo's haiku. Our agent will be caught in the download of the former, And that slow server will wait for a while. The best strategy is to close the connection and let the slow server go back to sleep.

Recall: Ref: ' Figure15 ', which shows the concept of synchronous program control flow. In that picture we can see that the function call is top-down and the exception is from the bottom up. If we want to cancel a synchronous call (which is just a hypothesis), the direction of the control flow is consistent with the direction of the function call. are from high-level to the bottom, 38 shows:

Figure 38: Synchronization program flow with hypothetical cancel operation

Of course, this is not possible in the Synchronizer, because the high-level code is not resumed before the end of the operation, naturally there is nothing to be canceled. However, in an asynchronous program, high-level code has control before the underlying code is completed, at least with the possibility of canceling its request before the underlying code finishes.

In the twisted program, the underlying request is contained in a Deferred object, which you can imagine as a "handle" to an external asynchronous operation. The normal flow of information in deferred is downward, from the underlying code to the high-level code, in the same direction as the flow of information returned in the synchronization program. Starting with twisted 10.1.0, high-level code can send back information in reverse-it can tell the underlying code that it no longer needs its results. 39:

Figure 39:deferredThe information flow in, including cancellation cancellationdeferreds

Let's look at some routines to see how the de- deferreds works. Note: In order to run these examples and the other code in this section, you will need to install twisted 10.1.0 or later. Consider deferred-cancel/defer-cancel-1.py:

From twisted.internet Import Deferdef callback (RES):    print ' callback Got: ', resd = defer. Deferred () D.addcallback (callback) D.cancel () print ' Done '

With the new cancellation feature, the Deferred class obtains a new method called Cancel . The above code creates a new Deferred, adds a callback, and then cancels the Deferred without firing it. The output is as follows:

doneunhandled Error in Deferred:traceback (most recent call last): Failure:twisted.internet.defer.CancelledError:

OK, canceling a deferred looks like making the error callback chain run, and the regular callback is not called at all. Also note that this error is: Twisted.internet.defer.CancelledError, one means Deferred The personalized exception that was canceled (but continue reading). Let's add an error callback, such as deferred-cancel/defer-cancel-2.py

From twisted.internet Import Deferdef callback (RES):    print ' callback Got: ', Resdef errback (err):    print ' Errback Got: ', errd = defer. Deferred () D.addcallbacks (callback, Errback) D.cancel () print ' Done '

Get the following output:

Errback got: [Failure instance:traceback (Failure with no frames): <class ' Twisted.internet.defer.CancelledError ' >:]done

So we can ' catch ' the error callback from cancel , just like any other deferred error.

OK, let's try to excite deferred and then cancel it, like deferred-cancel/defer-cancel-3.py:

From twisted.internet Import Deferdef callback (RES):    print ' callback Got: ', Resdef errback (err):    print ' Errback Got: ', errd = defer. Deferred () D.addcallbacks (callback, Errback) d.callback (' result ') d.cancel () print ' Done '

Here we use the conventional callback method to excite the deferred, and then cancel it. The output is as follows:

Callback Got:resultdone

Our callback is called (as we expected) and the program ends normally, just like cancel was not called at all. So canceling a deferred seems to have no effect at all if it has been fired (but read on!).

What happens if we fire it after canceling deferred ? See deferred-cancel/defer-cancel-4.py:

From twisted.internet Import Deferdef callback (RES):    print ' callback Got: ', Resdef errback (err):    print ' Errback Got: ', errd = defer. Deferred () D.addcallbacks (callback, Errback) D.cancel () d.callback (' result ') print ' done '

The output of this scenario is as follows:

Errback got: [Failure instance:traceback (Failure with no frames): <class ' Twisted.internet.defer.CancelledError ' >:]done

Interesting! As with the output of the second example, the deferredwas not fired at that time. So if deferred is canceled, it doesn't work. But why D.callback (' result ') did not produce an error, considering that it could not excite deferred More than once, why is the error callback chain not running?

Consider Figure39 again. Firing a deferred with a result or failure is the work of the underlying code, but canceling deferred is the behavior of high-level code. To inspire deferred means "This is your result", However, canceling deferred means "I don't want this result anymore." Remember that cancel is a new feature, so most of the existing twisted code does not handle the canceled operation. But twisted's developers canceled us. deferred 's ideas became possible, even including those written before twisted 10.1.0.

To achieve these ideas, the cancel method actually does two things:

1. Tell the Deferred object that you do not want that result if it has not yet returned (for example, Deferred is not fired), thus ignoring any subsequent calls to the callback or error callback.
2. At the same time, optionally, tell the underlying code that is producing the results what steps need to be taken to cancel the operation.

Since the old version of the twisted code is going to fire any deferredthat has been canceled, step#1 make sure our program doesn't break down if we cancel a deferredin an old library.

This means that we can cancel a deferredas we wish, and be sure that we won't get the result if it hasn't come yet (even those that are coming). But cancel deferred . The asynchronous operation may not be canceled. Terminating an asynchronous operation requires a context-specific action. You may need to close the network connection, rollback the database transaction, end the child process, and so on. Since deferred is merely a general purpose callback organizer, How does it know exactly what to do when you cancel it? Or, in other words, how does it pass the cancel request to the underlying code that first created and returned the deferred ? And I said:

I know, with a callback!

Essentially canceldeferreds

Well, first look at deferred-cancel/defer-cancel-5.py:

From twisted.internet import Deferdef Canceller (d):    print "I need to cancel this deferred:", Ddef Callback (RES):    p Rint ' callback got: ', Resdef errback (err):    print ' Errback got: ', errd = defer. Deferred (Canceller) # Created by Lower-level Coded.addcallbacks (callback, Errback) # Added by higher-level coded.cancel () print ' Done '

This example is basically the same as the second one, except for the third callback (Canceller). This callback was passed to it when we created the Deferred . Not added later. This callback is responsible for executing the context-sensitive actions required to terminate the asynchronous operation (of course, only if deferred is actually canceled). The Canceller callback is a necessary part of returning deferred 's underlying code, not the callback and error callbacks that receive deferred 's high-level code for itself.

Running this example will produce the following output:

I need to cancel the deferred: <deferred at 0xb7669d2cl>errback got: [Failure instance:traceback (Failure with no Frames): <class ' Twisted.internet.defer.CancelledError ' >:]done

As you can see, deferred that do not need to return results are passed to the Canceller callback. Here we do whatever we need to do to completely terminate the asynchronous operation. Note Canceller Called before the error callback chain is fired. In fact, we can choose to use any result or error in the cancel callback to fire deferred ourselves (this will take precedence over cancellederror failure). Both of these cases Described in deferred-cancel/defer-cancel-6.py and deferred-cancel/defer-cancel-7.py.

Make a simple test before firing reactor . We will use the Canceller callback to create a deferred, fire it normally, and then cancel it. You can be in Deferred-cancel See the code in/defer-cancel-8.py. By examining the output of that script, you will see that canceling a fired deferred does not call the Canceller callback. That's what we want, because there's nothing to cancel.

None of the examples we see at this time have actual asynchronous operations. Let's construct a simple program that invokes an asynchronous operation, and then we'll show you how to make that operation cancel. See Code deferred-cancel/defer-cancel-9.py:

From Twisted.internet.defer import Deferreddef send_poem (d):    print ' sending poem '    d.callback (' Once upon a Midnight Dreary ') def get_poem (): "" "    Return a poem 5 seconds later.    " " From twisted.internet import reactor    d = Deferred ()    reactor.calllater (5, Send_poem, D)    return ddef got_ Poem (poem):    print ' I got a poem: ', Poemdef poem_error (err):    print ' get_poem failed: ', errdef Main ():    from Twisted.internet Import reactor    Reactor.calllater (Ten, Reactor.stop) # Stop the reactor in seconds    d = Get_poe M ()    d.addcallbacks (Got_poem, Poem_error)    Reactor.run () main ()

This example contains a Get_poem function that uses the Calllater method of reactor to return a poem asynchronously after being called for 5 seconds. The main function calls Get_poem, adds a callback/error callback pair, and then starts reactor. We (also use calllater) arrange for reactor to stop after 10 seconds. Usually we add a callback to the deferred , But you'll soon know why we did it.

Running the program (after an appropriate delay) produces the following output:

Sending Poemi got a poem:once upon a midnight dreary

The program terminates after 10 seconds. Now try to cancel the deferredbefore the poem is sent. Simply add the following code to cancel after 2 seconds (before 5 seconds delay sending the poem):

Reactor.calllater (2, D.cancel) # Cancel after 2 seconds

For a complete example, see deferred-cancel/defer-cancel-10.py, which produces the following output:

Get_poem failed: [Failure instance:traceback (Failure with no frames): <class ' Twisted.internet.defer.CancelledError ' >:]sending poem

This example clearly shows that canceling a deferred does not cancel the asynchronous request behind it. 2 seconds later we saw the error callback output, which prints out the cancellederror error as we expected. But 5 seconds later, we saw it. The output of the Send_poem (but the callback on this deferred is not fired).

We are in the same situation as deferred-cancel/defer-cancel-4.py. " Cancel "The deferred is simply the end result being ignored, but it does not actually terminate the operation. As we have learned above, in order to get a truly deferred, you must add a cancel when it is created callback.

So what does this new callback need to do? Refer to the documentation for the Calllater method. Its return value is another object that implements the Idelayedcall , and with the cancel method We can prevent deferred calls from being executed.

This is very simple and the updated code is described in deferred-cancel/defer-cancel-11.py. All related changes are in the Get_poem function:

Def get_poem (): "" "    Return a poem 5 seconds later." " def Canceler (d):    # They don ' t want the poem anymore, so cancel the delayed call    Delayed_call.cancel ()    # at Thi S point we have three choices:    #   1. Do nothing, and the deferred would fire the Errback    #      chain with Cancellederror.    #   2. Fire the Errback chain with a different error.    #   3. Fire the callback chain with a alternative RESULT.D = Deferred (canceler) from twisted.internet import Reactordelayed_call = Reactor.calllater (5, Send_poem, D) return D

In this new version, we save the return value of Calllater so that it can be used in the cancel callback. The only job of the cancel callback is to call Delayed_call.cancel (). But as discussed earlier, we can choose to inspire custom deferred. The latest version of the program produces the following output:

Get_poem failed: [Failure instance:traceback (Failure with no frames): <class ' Twisted.internet.defer.CancelledError ';:]

As you can see, deferred was canceled and the asynchronous operation was really terminated (we don't see the output of send_poem).

Poetry Agency 3.0

As discussed in the introduction, the poetry proxy server is a good candidate for implementation cancellation, as this allows us to cancel the poetry download if it turns out that no one wants it (such as the client has closed the connection before we send the poem). Version 3.0 of the agent is located in twisted-server-4/ poetry-proxy.py, the deferred cancellation is realized. The change is first located in Poetryproxyprotocol:

Class Poetryproxyprotocol (Protocol):    def connectionmade (self):        self.deferred = self.factory.service.get_ Poem ()        self.deferred.addCallback (self.transport.write)        Self.deferred.addBoth (Lambda R: Self.transport.loseConnection ())   def connectionlost (self, Reason):       if self.deferred are not None:           Deferred, self.deferred = self.deferred, None           deferred.cancel () # Cancel the deferred if it hasn ' t fired

You can compare it with the old version. Two major changes are:

1. Save the deferredwe got from get_poem so that we can cancel it later if needed.
2. Cancels the deferredwhen the connection is closed. Note This operation will also be canceled deferred when we actually get the poem, but as we found in the previous case, cancel an excited deferred will not have any effect.

Now we need to make sure that canceling deferred will actually terminate the download of the poem. So we need to change Proxyservice:

Class Proxyservice (object):    poem = None # The cached poem    def __init__ (self, host, port):        self.host = host
   
    self.port = Port    def get_poem (self):        if self.poem are not None:           print ' Using cached poem. '           # return an already-fired deferred           return Succeed (self.poem)        def canceler (d):            print ' canceling poem Download. '            factory.deferred = None            connector.disconnect ()        print ' fetching poem from server. '        Deferred = deferred (Canceler)        deferred.addcallback (self.set_poem)        factory = Poetryclientfactory (deferred ) from        twisted.internet Import reactor        connector = reactor.connecttcp (Self.host, Self.port, Factory)        return factory.deferred    def set_poem (self, poem):        self.poem = poem        return poem
   

Again, you can compare it to the old version. This class has some new changes:

1. We save the return value of Reactor.connettcp, a Iconnector object. We can use the disconnect method on this object to close the connection.
2. We created the deferredwith the Canceler callback. The callback is a closure that uses connector to close the connection. But first you must set the Factory.deferred property to None. Otherwise, the factory fires deferred instead of cancellederror firing with a "connection shutdown" error callback. Since deferred has been canceled, it is more appropriate to Cancellederror .

You will also notice that we are creating deferred instead of poetryclientfactoryin proxyservice . Since the Canceler callback needs to acquire the iconnector object, proxyservice becomes the most convenient place to create deferred .

At the same time, as in our previous example, the Canceler callback is implemented as a closure. Closures appear to be useful in canceling the implementation of callbacks.

Let's try a new agent. Start a slow server first. It needs to be slow so that we have time to cancel:

Python blocking-server/slowpoetry.py--port 10001 poetry/fascination.txt

Now you can start the agent (remember you need twisted 10.1.0):

Python twisted-server-4/poetry-proxy.py--port 10000 10001

Now we can download a poem from the agent with any client, or use curl only:

Curl localhost:10000

After a few seconds, press ctrl-c to stop the client or curl process. Running the agent at the terminal you will see the following output:

Fetching poem from server. Canceling poem download.

You should see that the slow server has stopped printing the fragment of the verse it sent to the output because our agent is hung up. You can start and stop the client multiple times to verify that each download is canceled every time. But if you let the whole poem run, the agent caches it and sends it immediately after that. Another difficulty

More than once we have said that canceling a deferred that has been triggered is ineffective. However, this is not quite right. In:d oc: ' P13 ', we learned that a callback and error callback attached to a deferred may also return another deferred. In that case, the original (outer) deferred pauses to execute its callback chain and waits for the inner layer deferred to fire (see ' figure28 ' _).

Thus, even if a deferred fires a high-level code that emits an asynchronous request, it cannot receive the result because the callback chain pauses before waiting for the inner layer to finish deferred . So what happens when a high-level code cancels this external deferred ? In this case, the external deferred not only cancels itself (it has been fired); instead, the deferred cancels the internal deferred.

So when you cancel a deferred , you're probably not canceling the primary asynchronous operation, but some other asynchronous operation triggered by the former result. Call!

We can use an example to illustrate this. Consider code deferred-cancel/defer-cancel-12.py:

From twisted.internet import Deferdef cancel_outer (d):    print "outer cancel callback." def Cancel_inner (d):    print "inner cancel callback." def first_outer_callback (res):    print ' first outer callback, returning inner deferred '    return inner_ddef second_ Outer_callback (res):    print ' second outer callback got: ', Resdef outer_errback (err):    print ' outer errback got: ', Errouter_d = defer. Deferred (cancel_outer) inner_d = defer. Deferred (Cancel_inner) outer_d.addcallback (first_outer_callback) outer_d.addcallbacks (Second_outer_callback, Outer_errback) outer_d.callback (' result ') # at the outer deferred have fired, but are paused# on the inner deferred . print ' canceling outer deferred. ' Outer_d.cancel () print ' Done '

In this example, we created two deferred, outer, and inner, and an external callback returned to the internal deferred. First, we excite the external deferredand then cancel it. The output results are as follows:

First outer callback, returning inner deferredcanceling outer deferred.inner cancel Callback.outer errback got: [Failure i Nstance:traceback (failure with no frames): <class ' Twisted.internet.defer.CancelledError ' >:]done

As you can see, canceling an external deferred does not cause the external cancel callback to be fired. Instead, it cancels the internal deferred, so the internal cancel callback is fired, after which the external error callback receives the Cancellederror (from the internal deferred).

You may need to take a closer look at the code and make some changes to see how the results are affected.

Discuss

Canceling deferred is a very useful operation, so our program avoids doing the work that is not needed. However, as we have seen, it may be a little tricky.

An important fact to understand is that canceling a deferred does not mean canceling the asynchronous operation behind it. In fact, when writing this article, many deferreds will not be really "canceled", Because most of the twisted code is written before twisted 10.1.0 and has not been upgraded. This includes a lot of twisted itself apis! Check the document or source code to see if "Cancel deferred" really cancels the request behind it, or just ignore it.

The second important fact is that the deferred returned from your asynchronous APIs may not necessarily be canceled in the full sense. If you want to implement cancellation in your own program, you should first look at a number of examples in the twisted source code. cancellation is a new feature, so its patterns and best practices are still being developed.

Looking to the future

Now we have learned about all aspects of deferreds and the core concepts behind twisted. This means that we have nothing to introduce, because the rest of twisted mainly includes specific applications such as network programming or asynchronous database processing. So, in the next section, we want to take a detour and see the other two using asynchronous i/ O's system is similar to what twisted has in mind. Then, at the end, we'll make a package and suggest some ways to help you learn twisted.

Python Twisted Learning Series 19 (reprinted stulife The best Twisted Introductory tutorial)