The Python Twisted framework uses the Deferred object to manage the callback function.

The Python Twisted framework uses the Deferred object to manage the callback function.

First, we will throw some ideas about using callback programming:

• Activating errback is very important. Because errback has the same functions as other t blocks, you must ensure that they exist. They are not optional, but mandatory.
• Activation callback is not at the wrong time point as important as activation callback at the correct time point. Typically, callback and errback are mutually exclusive, that is, only one of them can be run.
• It is difficult to refactor the code using the callback function.

Deferred
Twisted uses the Deferred object to manage the sequence of callback functions. In some cases, a series of functions may be associated with the Deferred object to be called in order when asynchronous operations are completed (these series of callback functions are called callback function chains ); some functions are also called when exceptions occur in asynchronous operations. When the operation is completed, the first callback function is called first, or when an error occurs, the first callback function is called first, the Deferred object then passes the return values of each callback function or error handling callback function to the next function in the chain.
Callbacks
A twisted. internet. defer. Deferred object represents a function that will produce results at a certain time in the future. We can associate a callback function with a Deferred object. Once the result of this Deferred object is returned, this callback function will be called. In addition, the Deferred object allows developers to register an error handling callback function for it. The Deferred mechanism provides developers with standardized interfaces for various blocking or delayed operations.
From twisted. internet import reactor, defer

def getDummyData(inputData):  """  This function is a dummy which simulates a delayed result and  returns a Deferred which will fire with that result. Don't try too  hard to understand this.  """  print('getDummyData called')  deferred = defer.Deferred()  # simulate a delayed result by asking the reactor to fire the  # Deferred in 2 seconds time with the result inputData * 3  reactor.callLater(2, deferred.callback, inputData * 3)  return deferreddef cbPrintData(result):  """  Data handling function to be added as a callback: handles the  data by printing the result  """  print('Result received: {}'.format(result))deferred = getDummyData(3)deferred.addCallback(cbPrintData)# manually set up the end of the process by asking the reactor to# stop itself in 4 seconds timereactor.callLater(4, reactor.stop)# start up the Twisted reactor (event loop handler) manuallyprint('Starting the reactor')reactor.run()

Multiple callback Functions
Multiple callback functions can be associated with a Deferred object. The first callback function in the callback function chain is called with the result of the Deferred object as the parameter, the second callback function is called based on the result of the first function, and so on. Why do we need such a mechanism? Consider this situation, twisted. enterprise. adbapi returns a Deferred object-the result of an SQL query. A web window may add a callback function to the Deferred object to convert the query result to the HTML format, then, the Deferred object is passed forward. In this case, Twisted calls the callback function and returns the result to the HTTP client. In the case of an error or exception, the callback function chain will not be called.

from twisted.internet import reactor, deferclass Getter:  def gotResults(self, x):    """    The Deferred mechanism provides a mechanism to signal error    conditions. In this case, odd numbers are bad.    This function demonstrates a more complex way of starting    the callback chain by checking for expected results and    choosing whether to fire the callback or errback chain    """    if self.d is None:      print("Nowhere to put results")      return    d = self.d    self.d = None    if x % 2 == 0:      d.callback(x * 3)    else:      d.errback(ValueError("You used an odd number!"))  def _toHTML(self, r):    """    This function converts r to HTML.    It is added to the callback chain by getDummyData in    order to demonstrate how a callback passes its own result    to the next callback    """    return "Result: %s" % r  def getDummyData(self, x):    """    The Deferred mechanism allows for chained callbacks.    In this example, the output of gotResults is first    passed through _toHTML on its way to printData.    Again this function is a dummy, simulating a delayed result    using callLater, rather than using a real asynchronous    setup.    """    self.d = defer.Deferred()    # simulate a delayed result by asking the reactor to schedule    # gotResults in 2 seconds time    reactor.callLater(2, self.gotResults, x)    self.d.addCallback(self._toHTML)    return self.ddef cbPrintData(result):  print(result)def ebPrintError(failure):  import sys  sys.stderr.write(str(failure))# this series of callbacks and errbacks will print an error messageg = Getter()d = g.getDummyData(3)d.addCallback(cbPrintData)d.addErrback(ebPrintError)# this series of callbacks and errbacks will print "Result: 12"g = Getter()d = g.getDummyData(4)d.addCallback(cbPrintData)d.addErrback(ebPrintError)reactor.callLater(4, reactor.stop)reactor.run()

Note that the self. d method is processed in the gotResults method. This attribute is set to None before the Deferred object is activated due to a result or error, so that the Getter instance will no longer hold a reference to the Deferred object to be activated. This has several advantages. First, it can avoid the possibility that Getter. gotResults may repeatedly activate the same Deferred object (this will lead to an AlreadyCalledError exception ). In this way, a callback function that calls the Getter. getDummyData function can be added to the Deferred object without any problems. In addition, this makes it easier for the Python Garbage Collector to detect whether an object needs to be recycled through reference loops.
Visualized explanation
Write the image description here

1. The request method requests Data to the Data Sink to obtain the returned Deferred object.
2. The request method associates the callback function with the Deferred object.

1. When the result is ready, pass it to the Deferred object. If the operation succeeds, call the. callback (result) method of the Deferred object. If the operation fails, call the. errback (faliure) method of the Deferred object. Note that failure is an instance of the twisted. python. failure. Failure class.
2. The Deferred object uses result or faliure to activate the previously added callback function or error handling callback function. Then follow the following rules to continue the execution of the callback function chain:
The result of the callback function is always passed to the next callback function as the first parameter, forming a chained processor.
If a callback function throws an exception, it is transferred to the error handling callback function for execution.
If a faliure is not processed, it will be passed along the error processing callback function chain, which is a bit like the asynchronous version of the release T statement.
If an error handling callback function does not throw an exception or returns a twisted. python. failure. Failure instance, the callback function is executed.
Error Handling callback function
The error handling model of the Deferred object is based on Python exception handling. If no error occurs, all callback functions will be executed, one by one, as mentioned above.
If the error processing callback function is not executed (for example, an error occurs in the DB query), then a twisted. python. failure. the Failure object will be passed to the first error processing callback function (you can add multiple error processing callback functions, just like the callback function chain ). The error handling callback function chain can be used as the callback T code block in common Python code.
Unless an error is explicitly raise in the except T code block, the Exception object will be caught and no longer transmitted, and then the program will be executed normally. The same applies to the callback function chain for error handling. Unless you explicitly return a Faliure or throw an exception again, the error will stop spreading, then, the normal callback function chain will be executed from there (using the value returned by the error processing callback function ). If the error processing callback function returns a Faliure or throws an exception, the Faliure or exception will be passed to the next error processing callback function.
Note: if an error processing callback function does not return anything, it actually returns None, which means that the callback function chain will continue to be executed after the error processing callback function is executed. This may not be what you actually expected, so make sure that your error handling callback function returns a Faliure object (or the Faliure object passed to it as the parameter) or a meaningful return value to execute the next Callback Function.
Twisted. python. failure. Failure has a useful method called trap, which makes the following code a more efficient form:

try:  # code that may throw an exception  cookSpamAndEggs()except (SpamException, EggException):  # Handle SpamExceptions and EggExceptions  ...

Can be written:

def errorHandler(failure):  failure.trap(SpamException, EggException)  # Handle SpamExceptions and EggExceptionsd.addCallback(cookSpamAndEggs)d.addErrback(errorHandler)

If the parameter passed to faliure. trap does not match the error in Faliure, it will throw the error again.
Note that the functions of the twisted. internet. defer. Deferred. addCallbacks method are similar to those of the addCallback method, but they are not the same. Consider the following scenarios:

# Case 1d = getDeferredFromSomewhere()d.addCallback(callback1)    # Ad.addErrback(errback1)     # Bd.addCallback(callback2)d.addErrback(errback2)# Case 2d = getDeferredFromSomewhere()d.addCallbacks(callback1, errback1) # Cd.addCallbacks(callback2, errback2)

For Case 1, if an error occurs in callback1, errback1 will be called. For Case 2, errback2 is called.
Actually, in Case 1, Row A processes the successful execution of getDeferredFromSomewhere, and Row B processes errors that occur when getDeferredFromSomewhere is executed or when Row A's callback1 is executed. In Case 2, errback1 in Row C only processes errors generated when getDeferredFromSomewhere is executed, rather than Errors generated in callback1.
Unprocessed errors
If a Deferred object has an outstanding error (that is, if it has another errback, it will be called), it will be cleared by the garbage collector, twisted records the error traceback to the log file. This means that you may still be able to record errors without adding errback. But be careful if you still hold a reference to this Deferred object and it will never be cleared by the garbage collector, then you will never see this error (and your callbacks will never be executed mysteriously ). If you are not sure whether the above situation will happen, You should explicitly add an errback after the callback function chain, even if it is just like this:

# Make sure errors get loggedfrom twisted.python import logd.addErrback(log.err)

Process synchronous and asynchronous results
In some applications, synchronous functions and asynchronous functions may exist at the same time. For example, for a user authentication function, if it is from the memory to check whether the user has been authenticated, it can immediately return results; but if it needs to wait for data on the network, then it should return a Deferred object activated when the data arrives. That is to say, a function that wants to check whether the user has authenticated must be able to accept the results returned immediately and the Deferred object at the same time.
In the following example, authenticateUser uses isValidUser to authenticate the user:

def authenticateUser(isValidUser, user):  if isValidUser(user):    print("User is authenticated")  else:    print("User is not authenticated")

This function assumes that isValidUser is returned immediately. However, isValidUser may be authenticated asynchronously and return a Deferred object. It is possible to adjust this function to receive synchronous isValidUser and asynchronous isValidUser. At the same time, it is also possible to change the synchronous function to the return value as a Deferred object.
Process possible Deferred objects in Library Function Code
This is a synchronous user authentication method that may be passed to authenticateUser:

def synchronousIsValidUser(user):  '''  Return true if user is a valid user, false otherwise  '''  return user in ["Alice", "Angus", "Agnes"]

This is an Asynchronous User authentication method and returns a Deferred object:

from twisted.internet import reactor, deferdef asynchronousIsValidUser(user):  d = defer.Deferred()  reactor.callLater(2, d.callback, user in ["Alice", "Angus", "Agnes"])  return d

We initially wanted isValidUser to be synchronized to authenticateUser, but now we need to change it to an isValidUser implementation that can both Process Synchronization and process Asynchronization. You can use the maybeDeferred function to call isValidUser. This function ensures that the returned value of isValidUser is a Deferred object, even if isValidUser is a synchronous function:

from twisted.internet import deferdef printResult(result):  if result:    print("User is authenticated")  else:    print("User is not authenticated")def authenticateUser(isValidUser, user):  d = defer.maybeDeferred(isValidUser, user)  d.addCallback(printResult)

Now isValidUser can be both synchronous and asynchronous.
You can also rewrite the synchronousIsValidUser function to return a Deferred object. For more information, see here.
Cancel callback function
Motivation: a Deferred object may take a long time to call the callback function, and it will never be called. Sometimes you may not be so patient to wait for the result returned by Deferred. Since all the code to be executed after Deferred is completed is in your application or the called library, you can ignore the result when it takes a long time to receive the result. However, even if you choose to ignore this result, the underlying operations produced by this Deferred object still work in the background and consume machine resources, such as CPU time, memory, network bandwidth, and even disk capacity. Therefore, when the user closes the window, click the cancel button to disconnect from your server or send a "stop" command, in this case, you need to explicitly declare that you are no longer interested in the results of the previous operation, so that the original Deferred object can do some cleaning work and release resources.
This is a simple example. If you want to connect to an external machine, but this machine is too slow, you need to add a Cancel button in the application to terminate this connection attempt, so that you can connect to another machine. Here is the approximate logic of an application:

def startConnecting(someEndpoint):  def connected(it):    "Do something useful when connected."  return someEndpoint.connect(myFactory).addCallback(connected)# ...connectionAttempt = startConnecting(endpoint)def cancelClicked():  connectionAttempt.cancel()

Apparently, startConnecting is used by some UI elements to allow users to select the machine to connect. Then a Cancel button is accompanied to the cancelClicked function.
When connectionAttempt. cancel is called, the following operations are performed:

• Lead to potential connection operation termination, if it is still in progress
• In any case, the connectionAttempt Deferred object is completed in a timely manner.
• This may cause connectionAttempt, the Deferred object, to call the error processing function due to CancelledError errors.

Even if the cancellation operation already expresses the need to stop the underlying operation, it is unlikely that the underlying operation will immediately respond to this operation. Even in this simple example, there is an operation that will not be interrupted: domain name resolution, so it needs to be executed in a thread; connection operations in this application cannot be canceled while waiting for domain name resolution. Therefore, the Deferred object you want to cancel may not immediately call the callback function or the error handling callback function.

A Deferred object may wait for the completion of another Deferred object when executing any point of its callback function chain. There is no way to know whether everything is ready at a specific point in the callback function chain. Since many levels of functions in a callback function chain want to cancel the same Deferred object, functions of any level in the chain can be called at any time. cancel () function .. The cancel () function never throws any exception or returns any value. You can call it again, even if the Deferred object has been activated, it has no other callback functions.
While instantiating a Deferred object, you can provide it with a cancellation function (the Deferred object constructor is def _ init _ (self, canceller = None ): (source), the canceller can do anything. Ideally, everything it does will block the operations you requested, but this is not always guaranteed. Therefore, the cancellation of the Deferred object is only for the best effort. There are several reasons:
The Deferred object does not know how to cancel the underlying operation.
The underlying operation has been executed in a non-canceled state, because some irreversible operations may have been performed.
The Deferred object may already have results, so there is nothing to cancel.
After the cancel () function is called, no matter whether it can be canceled or not, the result will always be successful and no error will occur. In the first and second cases, because the underlying operations continue, the Deferred object can call its errback by taking twisted. internet. defer. CancelledError as the parameter.
If the canceled Deferred object is waiting for another Deferred object, the cancellation operation will be passed to this Deferred object.
You can refer to the API.
Default cancellation Behavior
All Deferred objects can be canceled, but only simple behaviors are provided, and no resources are released.
Consider the following example:

operation = Deferred()def x(result):  print("Hooray, a result:" + repr(x))operation.addCallback(x)# ...def operationDone():  operation.callback("completed")

If you need to cancel the Deferred object operation, but operation does not have a canceller cancellation function, one of the following two results will be generated:
If operationDone has been called, that is, the operation object has been completed, nothing will change. Operation still has a result, but since there are no other callback functions, there is no behavior change that can be seen.
If operationDone has not been called, operation will immediately activate errback with the CancelledError parameter.
Under normal circumstances, if a Deferred object has called the callback function and then calls callback, it will lead to an AlreadyCalledError. Therefore, callback can be called again on the canceled but no canceller Deferred object, which will only result in one empty operation. If you call callback multiple times, you will still get an AlreadyCalledError exception.
Create a Deferred object that can be canceled: Custom cancellation function
Assume that you are implementing an HTTP client and return a Deferred object that will be activated when the server returns a response. It is best to close the connection to cancel the connection. To cancel the function, you can pass a function as a parameter to the Deferred object constructor (this function is called when the Deferred object is canceled ):

class HTTPClient(Protocol):  def request(self, method, path):    self.resultDeferred = Deferred(      lambda ignore: self.transport.abortConnection())    request = b"%s %s HTTP/1.0\r\n\r\n" % (method, path)    self.transport.write(request)    return self.resultDeferred  def dataReceived(self, data):    # ... parse HTTP response ...    # ... eventually call self.resultDeferred.callback() ...

Now, if the cancel () function is called on the Deferred object returned by HTTPClient. request (), the HTTP request will be canceled (if not too late ). You must also call callback () on a canceled Deferred object with canceller ().
DeferredList
Sometimes you want to be notified after several different events, instead of every event. For example, you want to wait until all the connections in the list are closed. Twisted. internet. defer. DeferredList applies to this situation.
To create a DeferredList with multiple Deferred objects, you only need to pass a list of the Deferred objects you want to wait:
# Creates a DeferredList
Dl = defer. DeferredList ([deferred1, deferred2, deferred3])

Now we can regard this DeferredList as a common Deferred. For example, you can also call addCallbacks. This DeferredList calls its callback function only after all the Deferred objects are completed. The callback parameter is a list of all the returned results of the Deferred object contained in the DeferredList object. For example:

# A callback that unpacks and prints the results of a DeferredListdef printResult(result):  for (success, value) in result:    if success:      print('Success:', value)    else:      print('Failure:', value.getErrorMessage())# Create three deferreds.deferred1 = defer.Deferred()deferred2 = defer.Deferred()deferred3 = defer.Deferred()# Pack them into a DeferredListdl = defer.DeferredList([deferred1, deferred2, deferred3], consumeErrors=True)# Add our callbackdl.addCallback(printResult)# Fire our three deferreds with various values.deferred1.callback('one')deferred2.errback(Exception('bang!'))deferred3.callback('three')# At this point, dl will fire its callback, printing:#  Success: one#  Failure: bang!#  Success: three# (note that defer.SUCCESS == True, and defer.FAILURE == False)

Under normal circumstances, DeferredList does not call errback, but unless cousumeErrors is set to True, the errors generated in the Deferred object will still activate the errback of each Deferred object.
Note: If you want to apply the callback function to the Deferred object added to DeferredList, you must pay attention to the time when you add the callback function. Adding a Deferred object to DeferredList will also add a callback function to the Deferred object (when this callback function is running, its function is to check whether the DeferredList has been completed ). Most importantly, the callback function of the variable records the return value of the Deferred object and passes the value to the list that is finally handed over to the DeferredList callback function as a parameter.
Therefore, if you add a Deferred object to DeferredList and then add a callback function to the Deferred object, the returned values of the newly added callback function will not be passed to the callback function of DeferredList. To avoid this, we recommend that you do not add a callback function to the Deferred object after adding it to DeferredList.

def printResult(result):  print(result)def addTen(result):  return result + " ten"# Deferred gets callback before DeferredList is createddeferred1 = defer.Deferred()deferred2 = defer.Deferred()deferred1.addCallback(addTen)dl = defer.DeferredList([deferred1, deferred2])dl.addCallback(printResult)deferred1.callback("one") # fires addTen, checks DeferredList, stores "one ten"deferred2.callback("two")# At this point, dl will fire its callback, printing:#   [(True, 'one ten'), (True, 'two')]# Deferred gets callback after DeferredList is createddeferred1 = defer.Deferred()deferred2 = defer.Deferred()dl = defer.DeferredList([deferred1, deferred2])deferred1.addCallback(addTen) # will run *after* DeferredList gets its valuedl.addCallback(printResult)deferred1.callback("one") # checks DeferredList, stores "one", fires addTendeferred2.callback("two")# At this point, dl will fire its callback, printing:#   [(True, 'one), (True, 'two')]

DeferredList uses three keyword parameters to customize its behavior: fireOnOneCallback, fireOnOneErrback, and cousumeErrors. If fireonecallback is set, as long as a Deferred object calls its callback function, DeferredList will immediately call its callback function. Similarly, if fireOnOneErrback is set, Deferred calls errback and DeferredList calls its errback. Note that DeferredList is only one-time, so after a callback or errback call, it will do nothing (it will ignore all the results its Deferred passes to it ).
The fireOnOneErrback option is useful when you want to wait for all tasks to be successfully executed and need to know immediately when an error occurs.
The consumeErrors parameter will cause errors in the Deferred object contained in DeferredList. After a DeferredList object is created, it will not be passed to the errbacks of the original Deferred object. After a DeferredList object is created, any errors generated in a single Deferred object will be converted into a callback call with the result of None. This option can be used to prevent the "Unhandled error in Deferred" Warning in Deferred contained in it, instead of adding additional errbacks (otherwise, you need to add errback for each Deferred object to eliminate this warning ). Passing a True value to the consumeErrors parameter does not affect the behavior of fireOnOneCallback and fireOnOneErrback. Always use this parameter unless you want to add callbacks or errbacks to the Deferred objects in these lists in the future, or unless you know that they do not produce errors. Otherwise, an "unhandled error" in the log is recorded by Twisted ".
A common use of DeferredList is to combine the results of some parallel asynchronous operations. If all operations are successful, the operation is successful. If one operation fails, the operation fails. Twisted. internet. defer. gatherResults is a shortcut:

from twisted.internet import deferd1 = defer.Deferred()d2 = defer.Deferred()d = defer.gatherResults([d1, d2], consumeErrors=True)def cbPrintResult(result):  print(result)d.addCallback(cbPrintResult)d1.callback("one")# nothing is printed yet; d is still awaiting completion of d2d2.callback("two")# printResult prints ["one", "two"]

Chained Deferred
If you need a Deferred object to wait for the execution of another Deferred object, all you have to do is return a Deferred object from the callback function in its callback function chain. Specifically, if you return the Deferred object B from A callback function of A Deferred object A, the callback function chain of A will wait before calling the callback () function of B. In this case, the first parameter of the next Callback Function of A is the result returned by the last callback function of B.
Note: If a 'referred' object directly or indirectly returns it in its callback function, such behavior is not defined. The code will try to detect this situation and then give a warning. Exceptions may be thrown directly in the future.
If this looks complicated, don't worry-when you encounter this situation, you may recognize it directly and know why it produces such a result. If you need to manually set the Deferred object
Link up. Here is a convenient method:

chainDeferred(otherDeferred)

Summary
We realized how deferred helped us solve these problems:
We cannot ignore errback and need it in any asynchronous programming API. Deferred supports errbacks.
Multiple activation callbacks may cause serious problems. Deferred can only be activated once, which is similar to try/try t in Synchronous Programming.
Programs with Callbacks are quite difficult to refactor. With deferred, we can modify the callback chain to reconstruct the program.