Python's twisted framework uses deferred objects to manage callback functions

Let's start with some of the ideas we've discussed when using callback programming:

Activating the Errback is very important. Because the Errback function is the same as the except block, users need to ensure that they exist. They are not optional, but must be options.

It is equally important not to activate the callback at the wrong point in time and to activate the callback at the correct point in time. The typical usage is that callback and Errback are mutually exclusive and can only run one of them.

Code that uses a callback function is somewhat difficult to refactor.

Deferred
Twisted uses the deferred object to manage the sequence of callback functions. In some cases, you might want to associate a series of functions to a deferred object so that it can be called sequentially when the asynchronous operation completes (these series of callback functions are called callback function chains), and some functions are called when an asynchronous operation is unexpected. When the operation completes, the first callback function is called first, or when the error occurs, the first error handling callback function is called first, and then the deferred object passes the return value of each callback function or error handling callback function to the next function in the chain.
Callbacks
A Twisted.internet.defer.Deferred object represents a function that is bound to produce a result at some point in the future. We can associate a callback function to the deferred object, and once the deferred object has a result, the callback function is called. In addition, the deferred object allows the developer to register an error-handling callback function for it. The deferred mechanism provides the developer with a standardized interface for a wide variety of blocking or delay operations.
From twisted.internet import reactor, defer

def getDummyData(inputData):
  """
  This function is a dummy which simulates a delayed result and
  returns a Deferred which will fire with that result. Don't try too
  hard to understand this.
  """
  print('getDummyData called')
  deferred = defer.Deferred()
  # simulate a delayed result by asking the reactor to fire the
  # Deferred in 2 seconds time with the result inputData * 3
  reactor.callLater(2, deferred.callback, inputData * 3)
  return deferred
 
def cbPrintData(result):
  """
  Data handling function to be added as a callback: handles the
  data by printing the result
  """
  print('Result received: {}'.format(result))
 
deferred = getDummyData(3)
deferred.addCallback(cbPrintData)
 
# manually set up the end of the process by asking the reactor to
# stop itself in 4 seconds time
reactor.callLater(4, reactor.stop)
# start up the Twisted reactor (event loop handler) manually
print('Starting the reactor')
reactor.run()

Multiple callback functions
can be associated with multiple callback functions on a deferred object, and the first callback function on the chain of the callback function is invoked with the result of the deferred object, and the second callback function is invoked with the result of the first function, and so on. Why do we need such a mechanism? Considering the situation, TWISTED.ENTERPRISE.ADBAPI returns a deferred object-the result of an SQL query, and there may be a web window that adds a callback function to the deferred object. To convert the query results into HTML and then move the deferred object forward, twisted calls the callback function and returns the result to the HTTP client. In the event of an error or exception, the callback function chain is not called.

from twisted.internet import reactor, defer
 
 
class Getter:
  def gotResults(self, x):
    """
    The Deferred mechanism provides a mechanism to signal error
    conditions. In this case, odd numbers are bad.
 
    This function demonstrates a more complex way of starting
    the callback chain by checking for expected results and
    choosing whether to fire the callback or errback chain
    """
    if self.d is None:
      print("Nowhere to put results")
      return
 
    d = self.d
    self.d = None
    if x % 2 == 0:
      d.callback(x * 3)
    else:
      d.errback(ValueError("You used an odd number!"))
 
  def _toHTML(self, r):
    """
    This function converts r to HTML.
 
    It is added to the callback chain by getDummyData in
    order to demonstrate how a callback passes its own result
    to the next callback
    """
    return "Result: %s" % r
 
  def getDummyData(self, x):
    """
    The Deferred mechanism allows for chained callbacks.
    In this example, the output of gotResults is first
    passed through _toHTML on its way to printData.
 
    Again this function is a dummy, simulating a delayed result
    using callLater, rather than using a real asynchronous
    setup.
    """
    self.d = defer.Deferred()
    # simulate a delayed result by asking the reactor to schedule
    # gotResults in 2 seconds time
    reactor.callLater(2, self.gotResults, x)
    self.d.addCallback(self._toHTML)
    return self.d
 
 
def cbPrintData(result):
  print(result)
 
 
def ebPrintError(failure):
  import sys
  sys.stderr.write(str(failure))
 
 
# this series of callbacks and errbacks will print an error message
g = Getter()
d = g.getDummyData(3)
d.addCallback(cbPrintData)
d.addErrback(ebPrintError)
 
# this series of callbacks and errbacks will print "Result: 12"
g = Getter()
d = g.getDummyData(4)
d.addCallback(cbPrintData)
d.addErrback(ebPrintError)
 
reactor.callLater(4, reactor.stop)
reactor.run()

One thing to note is how the SELF.D is handled in method Gotresults. This property is set to none until the deferred object is either result or error activated, so that the getter instance will no longer hold a reference to the deferred object that will be activated. There are several advantages to doing so, first of all, this avoids the possibility that getter.gotresults sometimes repeats the same deferred object (which can cause a Alreadycallederror exception). Second, this makes it possible to add a callback function that calls the Getter.getdummydata function on the deferred object without causing the problem. Also, this makes it easier for the Python garbage collector to detect whether an object needs to be recycled by referencing the loop.
Explanation of the visualization
Write a picture description here

1. Request the method request data to Sink, get the returned deferred object.
2. The request method associates the callback function to the deferred object.

1. When the result is ready, pass it to the deferred object. If the operation succeeds, the. Callback (Result) method of the deferred object is called, and if the operation fails, the. Errback (Faliure) method of the deferred object is called. Note that failure is an instance of the Twisted.python.failure.Failure class.
2.Deferred objects use result or faliure to activate previously added callback functions or error handling callback functions. Then follow the following rules to continue execution along the callback chain:
The result of the callback function is always passed to the next callback function as the first argument, thus forming a chain-like processor.
If a callback function throws an exception, it goes to the error handling callback function to execute.
If a faliure is not processed, it is passed along the error-handling callback function chain, which is somewhat like an asynchronous version of the except statement.
If an error handling callback function does not throw an exception or returns an Twisted.python.failure.Failure instance, then go to execute the callback function.
Error Handling callback function
The error-handling model of the deferred object is based on Python exception handling. In the absence of an error, all the callback functions are executed, one after the other, as mentioned above.
If you do not execute a callback function but execute an error-handling callback function (such as a DB query that has an error), Then a Twisted.python.failure.Failure object is passed to the first error-handling callback function (you can add multiple error-handling callback functions, just like the callback function chain). The error handling callback function chain can be used as a except code block in normal Python code.
Unless an error is explicitly raise in the except code block, the exception object is captured and no longer propagated, and then the program is executed normally. The same is true for error-handling callback function chains, unless you explicitly return a faliure or re-throw an exception, the error stops propagating, and then the normal callback chain is executed from there (using the value returned by the error handling callback function). If the error handling callback function returns a faliure or throws an exception, the Faliure or exception is passed to the next error handling callback function.
Note that if an error-handling callback function returns Nothing, it actually returns none, which means that the callback function chain will continue to execute after the error handling callback function executes. This may not be what you actually expect, so make sure your error handling callback function returns a Faliure object (or the Faliure object passed to it as a parameter) or a meaningful return value to execute the next callback function.
Twisted.python.failure.Failure has a useful method called trap, which can make the following code another form of efficiency:

try:
  # code that may throw an exception
  cookSpamAndEggs()
except (SpamException, EggException):
  # Handle SpamExceptions and EggExceptions
  ...

Can be written as:

def errorHandler(failure):
  failure.trap(SpamException, EggException)
  # Handle SpamExceptions and EggExceptions
 
d.addCallback(cookSpamAndEggs)
d.addErrback(errorHandler)

If the argument passed to Faliure.trap does not match the error in the Faliure, it will re-throw the error.
There is also a need to note that the function and Addcallback of the Twisted.internet.defer.Deferred.addCallbacks method are similar to the Adderrback function, but not exactly the same. Consider the following scenario:

# Case 1
d = getDeferredFromSomewhere()
d.addCallback(callback1)    # A
d.addErrback(errback1)     # B
d.addCallback(callback2)
d.addErrback(errback2)
 
# Case 2
d = getDeferredFromSomewhere()
d.addCallbacks(callback1, errback1) # C
d.addCallbacks(callback2, errback2)

For Case 1, if an error occurs inside the Callback1, then Errback1 is called. For Case 2, the call is Errback2.
In fact, in case 1, row a handles the success of getdeferredfromsomewhere execution, and row B handles the error that occurs when Getdeferredfromsomewhere executes or the callback1 of row a executes. In case 2, the Errback1 in line C only handles errors generated by getdeferredfromsomewhere execution, not the errors generated in Callback1.
Unhandled error
If a deferred object is cleared by the garbage collector when there is an unhandled error (that is, if it still has the next errback), twisted will log the error traceback to the log file. This means that you may still be able to log errors without adding errback. Be careful, however, that if you still have a reference to the deferred object and it will never be cleaned up by the garbage collector, you will never see the error (and your callbacks will never be executed mysteriously). If you are unsure whether this will happen, you should explicitly add a errback after the callback function chain, even if this is the case:

# Make sure errors get logged
from twisted.python import log
d.addErrback(log.err)

Working with synchronous and asynchronous results
In some applications, there may be simultaneous functions and asynchronous functions. For example, for a user authentication function, if it checks whether the user is authenticated from memory, it can return the result immediately, but if it needs to wait for data on the network, it should return a deferred object that is activated when the data arrives. This means that a function that wants to check whether a user has been authenticated needs to be able to accept both the result of immediate return and the deferred object.
In the following example, Authenticateuser uses Isvaliduser to authenticate the user:

def authenticateUser(isValidUser, user):
  if isValidUser(user):
    print("User is authenticated")
  else:
    print("User is not authenticated")

This function assumes that Isvaliduser is returned immediately, however the Isvaliduser may actually be an asynchronous authenticated user and return a deferred object. It is possible to adjust this function to be able to receive both synchronous Isvaliduser and receiving asynchronous Isvaliduser. It is also possible to change the synchronous function to the return value of the deferred object.
Handling possible deferred objects in library function code
This is a user authentication method that may be passed to Authenticateuser for synchronization:

def synchronousIsValidUser(user):
  '''
  Return true if user is a valid user, false otherwise
  '''
  return user in ["Alice", "Angus", "Agnes"]

This is an asynchronous user authentication method that returns a deferred object:

from twisted.internet import reactor, defer
 
def asynchronousIsValidUser(user):
  d = defer.Deferred()
  reactor.callLater(2, d.callback, user in ["Alice", "Angus", "Agnes"])
  return d

Our initial implementation of Authenticateuser wanted Isvaliduser to be synchronous, but now we need to change it to a isvaliduser implementation that can handle both synchronization and asynchronous processing. For this, you can use the Maybedeferred function to call Isvaliduser, which guarantees that the return value of the Isvaliduser function is a deferred object, even if Isvaliduser is a synchronous function:

from twisted.internet import defer
 
def printResult(result):
  if result:
    print("User is authenticated")
  else:
    print("User is not authenticated")
 
def authenticateUser(isValidUser, user):
  d = defer.maybeDeferred(isValidUser, user)
  d.addCallback(printResult)

Now the Isvaliduser can be both synchronous and asynchronous.
You can also rewrite the Synchronousisvaliduser function to return a deferred object, which you can refer to here.
Cancel Callback function
Motivation: A deferred object may take a long time to invoke a callback function, or even never be called. Sometimes you may not have the patience to wait for deferred to return the results. Now that all the code to execute after deferred is done is in your app or in the library that you call, you can choose to ignore the result when it has been a long time before you receive the result. However, even if you choose to ignore this result, the underlying operation that the deferred object produces still works in the background and consumes machine resources such as CPU time, memory, network bandwidth, and even disk capacity. Therefore, when the user closes the window, clicks the Cancel button, disconnects from your server, or sends a "stop" instruction, you need to explicitly state that you are no longer interested in the results of the previously scheduled operation, so that the original deferred object can do some cleanup work and release resources.
This is a simple example, you want to connect to an external machine, but this machine is too slow, so you need to add a Cancel button in the app to terminate this connection attempt so that the user can connect to another machine. Here is the approximate logic of such an application:

def startConnecting(someEndpoint):
  def connected(it):
    "Do something useful when connected."
  return someEndpoint.connect(myFactory).addCallback(connected)
# ...
connectionAttempt = startConnecting(endpoint)
def cancelClicked():
  connectionAttempt.cancel()

Obviously, startconnecting is used by some UI elements to let the user choose which machine to connect to. Then there is a Cancel button accompanying the cancelclicked function.
When Connectionattempt.cancel is called, the following actions occur:

Causes a potential connection operation to terminate if it is still in progress

Whatever it is, it makes Connectionattempt the deferred object to be completed in time.

It is possible to cause connectionattempt this deferred object to invoke the error handler because of a cancellederror error

Even though this cancellation has already expressed the need to stop the underlying operation, the underlying operation is unlikely to respond immediately. Even in this simple example there is an operation that will not be interrupted: domain name parsing, so it needs to be executed in a thread, and the connection operation in this application cannot be canceled while waiting for the domain name to resolve. So the deferred object you want to cancel may not immediately call the callback function or the error handling callback function.

A deferred object may wait for the completion of another deferred object at any point in the chain that executes its callback function. There is no way to know at a particular point in the callback function chain whether everything is ready. Since it is possible that functions at many levels of a callback function chain will want to cancel the same deferred object, any function at any level of the chain may be called at any time. The Cancel () function: the Cancel () function never throws any exceptions or returns any values. You can call it repeatedly, even if the deferred object has already been activated, and it has no remaining callback functions.
When instantiating a deferred object, you can give it a cancellation function (the deferred object's constructor is Def __init__ (self, Canceller=none): (source)), This canceller can do anything. Ideally, everything it does will block the action you requested before, but it's not always guaranteed. So the cancellation of the deferred object is just the best effort. There are several reasons: the
Deferred object does not know how to cancel the underlying operation. The
underlying operation has been performed in a non-canceled state because some irreversible operations may have been performed. The
deferred object may already have results, so there is nothing to cancel.
After calling the Cancel () function, the result will always be successful, regardless of whether it can be canceled, without any error. In the first and second and the case, because the underlying operation is still continuing, the deferred object can be twisted.internet.defer.CancelledError as a parameter to invoke its errback.
If the canceled deferred object is waiting for another deferred object, the cancel operation is passed forward to the deferred object. The
can refer to the API.
Default Cancel Behavior
All deferred objects support cancellation, but only provide simple behavior and no resources are freed.
Consider the following example:

operation = Deferred()
def x(result):
  print("Hooray, a result:" + repr(x))
operation.addCallback(x)
# ...
def operationDone():
  operation.callback("completed")

If you need to cancel operation this deferred object, and operation does not have a canceller cancellation function, it will produce one of the following two results:
If Operationdone has been called, that is, the Operation object has been completed, nothing will change. Operation still has a result, but since there is no other callback function, there is no action to see the change.
If Operationdone has not been called yet, operation will immediately activate Errback with Cancellederror as a parameter.
Under normal circumstances, if a deferred object has called the callback function again to call callback it will result in a alreadycallederror. Therefore, callback can be called again on a deferred object that has been canceled but not canceller, which results in an empty operation only. If you call callback multiple times, you will still get a Alreadycallederror exception.
Create a deferred object that can be canceled: custom Cancel function
Suppose you are implementing an HTTP client that returns a deferred object that will be activated when the server returns a response. Cancellation is best to close the connection. In order for the cancellation function to do this, you can pass a function as a parameter to the constructor of the deferred object (this function is called when the deferred object is canceled):

class HTTPClient(Protocol):
  def request(self, method, path):
    self.resultDeferred = Deferred(
      lambda ignore: self.transport.abortConnection())
    request = b"%s %s HTTP/1.0\r\n\r\n" % (method, path)
    self.transport.write(request)
    return self.resultDeferred
 
  def dataReceived(self, data):
    # ... parse HTTP response ...
    # ... eventually call self.resultDeferred.callback() ...

Now if the Cancel () function is called on the deferred object returned by Httpclient.request (), the HTTP request is canceled (if it's not too late). Note that callback () is also called on a deferred object that has been canceled with Canceller.
Deferredlist
Sometimes you want to be notified when a few different events occur, not every event. For example, you want to wait for all connections in a list to close. Twisted.internet.defer.DeferredList is suitable for this situation.
To create a deferredlist with multiple deferred objects, simply pass in a list of deferred objects you want to wait for:
# Creates a deferredlist
DL = defer. Deferredlist ([Deferred1, Deferred2, Deferred3])

Now you can think of this deferredlist as an ordinary deferred, such as you can call addcallbacks and so on. The deferredlist will call its callback function after all the deferred objects have been completed. The parameter of this callback function is a list of the results returned by all deferred objects contained in this Deferredlist object, for example:

# A callback that unpacks and prints the results of a DeferredList
def printResult(result):
  for (success, value) in result:
    if success:
      print('Success:', value)
    else:
      print('Failure:', value.getErrorMessage())
 
# Create three deferreds.
deferred1 = defer.Deferred()
deferred2 = defer.Deferred()
deferred3 = defer.Deferred()
 
# Pack them into a DeferredList
dl = defer.DeferredList([deferred1, deferred2, deferred3], consumeErrors=True)
 
# Add our callback
dl.addCallback(printResult)
 
# Fire our three deferreds with various values.
deferred1.callback('one')
deferred2.errback(Exception('bang!'))
deferred3.callback('three')
 
# At this point, dl will fire its callback, printing:
#  Success: one
#  Failure: bang!
#  Success: three
# (note that defer.SUCCESS == True, and defer.FAILURE == False)

Normally deferredlist does not call Errback, but unless Cousumeerrors is set to true, errors generated in deferred objects will still activate the deferred of each Errback object.
Note that if you want to apply a callback function to a deferred object that you add to deferredlist, you need to be aware of the timing of adding the callback function. Adding a deferred object to deferredlist causes a callback function to be added to the deferred object at the same time (when the callback function runs, its function is to check whether the Deferredlist is complete). Most importantly, the variable's callback function records the return value of the deferred object and passes the value to the Deferredlist callback function as the list of arguments.
Therefore, if you add a callback function to the deferred object after adding a deferred to Deferredlist, the return value of the newly added callback function is not passed to the Deferredlist callback function. To avoid this, it is recommended not to add a callback function to the deferred after adding a deferred object to the Deferredlist.

def printResult(result):
  print(result)
 
def addTen(result):
  return result + " ten"
 
# Deferred gets callback before DeferredList is created
deferred1 = defer.Deferred()
deferred2 = defer.Deferred()
deferred1.addCallback(addTen)
dl = defer.DeferredList([deferred1, deferred2])
dl.addCallback(printResult)
deferred1.callback("one") # fires addTen, checks DeferredList, stores "one ten"
deferred2.callback("two")
# At this point, dl will fire its callback, printing:
#   [(True, 'one ten'), (True, 'two')]
 
# Deferred gets callback after DeferredList is created
deferred1 = defer.Deferred()
deferred2 = defer.Deferred()
dl = defer.DeferredList([deferred1, deferred2])
deferred1.addCallback(addTen) # will run *after* DeferredList gets its value
dl.addCallback(printResult)
deferred1.callback("one") # checks DeferredList, stores "one", fires addTen
deferred2.callback("two")
# At this point, dl will fire its callback, printing:
#   [(True, 'one), (True, 'two')]

Deferredlist accepts three keyword parameters to customize its behavior: Fireononecallback, Fireononeerrback, and Cousumeerrors. If Fireononecallback is set, the callback function of Deferredlist is immediately invoked whenever a deferred object invokes its callback function. Similarly, if Fireononeerrback is set, the errback of Errback,deferredlist is invoked whenever a deferred is called. Note that Deferredlist is just a one-time, so after a callback or Errback call, it does nothing (it ignores all the results that its deferred passes to it). The
Fireononeerrback option is useful when you want to wait for everything to execute successfully, and you need to know it immediately when something goes wrong. The
consumeerrors parameter causes errors in deferred objects contained in deferredlist to be passed to the deferred of each Errbacks object after the Deferredlist object has been established. After the Deferredlist object has been created, any errors generated in any single deferred object will be converted to a callback call with a result of none. Use this option to prevent the "unhandled error in Deferred" warning in the Deferred it contains without adding additional errbacks (otherwise, to eliminate this warning you need to add Deferred for each Errback object). Passing a true to the Consumeerrors parameter does not affect the behavior of fireononecallback and Fireononeerrback. You should always use this parameter unless you want to add callbacks or Errbacks to the deferred objects in these lists in the future, or unless you know they don't produce an error. Otherwise, an error will result in a "unhandled error" that is twisted logged to the log. A common usage of the
Deferredlist is to combine some parallel asynchronous operation results together. If all the operations are successful, the operation succeeds and if one fails, the operation fails. Twisted.internet.defer.gatherResults is a shortcut:

from twisted.internet import defer
d1 = defer.Deferred()
d2 = defer.Deferred()
d = defer.gatherResults([d1, d2], consumeErrors=True)
 
def cbPrintResult(result):
  print(result)
 
d.addCallback(cbPrintResult)
 
d1.callback("one")
# nothing is printed yet; d is still awaiting completion of d2
d2.callback("two")
# printResult prints ["one", "two"]

Chain-Deferred
If you need a deferred object to wait for the execution of another deferred object, all you have to do is return a deferred object from the callback function in its callback function chain. Specifically, if you return deferred object B from a callback function of a deferred object A, then A's callback function chain waits before the callback () function Call of B. At this point, the first parameter of the next callback function of a is the result returned by the last callback function of B.
Note that if a ' Deferred ' object returns itself directly or indirectly in its callback function, then this behavior is undefined. The code tries to detect the situation and then gives a warning. Exceptions may be thrown directly in the future.
If this looks a little complicated, don't worry--when you're in this situation, you may recognize it and know why. If you need to manually put the deferred object
Link up, here's a handy way:

Chaindeferred (otherdeferred)

Summary
We recognize how deferred helps us solve these problems:
We can't ignore errback, we need it in any asynchronous programming API. Deferred supports Errbacks. The
activation callback multiple times can cause serious problems. Deferred can only be activated once, which is similar to the processing method of try/except in synchronous programming. The
program with callbacks is very difficult to refactor. With deferred, we refactor the program by modifying the callback chain.