Tornado Study Record II

Coroutines

Coroutines is the recommended-to-write asynchronous code in Tornado. Coroutines Use the Python yield keyword to suspend and resume execution instead of a chain in callbacks (cooperative light Weight threads as seen in frameworks like Gevent is sometimes called coroutines as well, but in Tornado all coroutines us E Explicit context switches and is called as asynchronous functions).

Tornado recommended use of the association. The process uses the yield keyword to suspend and restart execution instead of using chained callbacks. Then mentioned Gevent, Gevent also implemented the co-process, but the principle and tornado are not the same. So, what the hell is the process? Do not understand the definition of Wikipedia, I made one, I do not know what is the association, I know that the process can be asynchronous, can be executed under a thread of a lot of concurrency, and each concurrency is a shared context, I do not have to worry about whether a data structure is thread-safe, do not care about data synchronization is not synchronized, Lock is not a lock problem. Generally speaking, the association is related to these.

Gevent use lightweight threads to achieve the above functions, lightweight threads, it seems to be the system-level thread subdivision, when encountered asynchronous and high concurrency, or multithreaded that set, but she can manage it, we do not care, we see a single thread, synchronous But performance is like Async.

Tornado uses context switches contextual switching, imagining high concurrency scenarios, which can be switched between each concurrency, and generally not understood as the event loop that drives context switching.

Last year, even saw a simple event loop demo source code, and then confused forced, I can understand the event loop, but who to do Io, who to do the notification? Definitely not an event loop for this thread, is it? Using yield switching, the principle is cumbersome, I can generally understand, but who inform the yield? Find a lot of articles, brain capacity is limited, feel not to say clearly.

Coroutines is almost as simple as synchronous code, but without the expense of a thread. They also make concurrency easier to reason on by reducing the number of places where a context switch can happen.

The code for the co-process is almost as simple as synchronizing the code, but without the overhead of the thread. And the co-process makes concurrency easier because it reduces context-switching scenarios.

Review the Code of the co-process again

 from  tornado import   Gen@gen.coroutine  def   Fetch_coroutine (URL): http_client  = Asynchttpclient () response  = yield   Http_client.fetch (URL)  #   #   generator is not allowed and you must use  #   raise Gen.    Return (response.body)  #   instead.  return  response.body 

Skipped Python 3.5 async, await, temporarily still not used, and the heart has a shadow.

How it works

A function containing is yield a generator. All generators is asynchronous; When called they return a generator object instead of running to completion. @gen.coroutinethe decorator communicates yield with the generator via the expressions, and with the Coroutine ' s caller by Returnin G A Future .

A function that contains yield becomes the generator. All generators are asynchronous, and when the generator is called, it returns a generator object instead of executing to the end. @gen. Coroutine adorner interacts with the generator through yield and returns the caller a future

It is now possible to speculate that the person behind the job is @gen. Coroutine, when IO is finished, it is to be notified by yield, but there is no intuitive sense for context switch.

Simplified example of a generator inner loop

# simplified inner loop of Tornado.gen.Runner def Run (self):     # Send (x) makes the current yield return x.    # It returns when the next yield is reached    Future = Self.gen.send (self.next)    def  Callback (f):        = F.result ()        Self.run ()    Future.add_done_callback (callback)

The decorator receives A future  from The generator, waits (without blocking) for that  future  to complete, then " Unwraps "The future  and Sends the result back into the generator as the result of The yield  expression. Most asynchronous code never touches The  Future  class directly except to immediately pass The future  returned by a asynchronous function to A yield  expression.

The adorner receives a future from the generator, waits for the future execution to end without blocking, then splits the future and returns the execution result as the result of the yield expression back to the generator. Most asynchronous code does not manipulate the future class directly unless the future object returned by the asynchronous function is passed directly to the yield expression.

Now, in retrospect, I was out of the way, trying to figure out what the above code meant. Then look at the source code, looking for a blog, but forget the original purpose is simply to use Tornado write service only.

How to call a Coroutine

In fact, there is no coroutine of the principle of the involved, just calmly tell you, I have a method of this, you how to use.

Coroutines don't raise exceptions in the normal Way:any exception they raise'll be trapped in the Future until it's Yie Lded. This means it's important to call coroutines in the right-of-the-the-right-of-the-go unnoticed:

This is to say that Coroutine should be written to generate the exception correctly, because the exception will be wrapped in the future object until it is yielded. In other words, if there is an exception, not being yielded is out of the way.

  @gen. coroutine  def   divide (x, y):  return  x/ y  def   Bad_call ():  #   should raise a zerodivisionerror, but it won ' t because  #   The coroutine is called incorrectly.  Divide (1, 0) 

In nearly all cases, any function that calls a coroutine must is a coroutine itself, and use the keyword on the call yield  . When you were overriding a method defined in a superclass, consult the documentation to see if Coroutines is allowed (the Documentation should say that the method ' may be a coroutine ' or ' may return a Future '):

In almost all scenarios, any function that calls Coroutine must be a coroutine and is invoked with yield. As stated above, yield will disassemble the future object and return result.

When you overload the method defined in the superclass, refer to whether the document allows this method to be overloaded into Coroutine, the document will probably say, this method can be a coroutine, or it can be a future object

@gen. Coroutine def Good_call ():     # yield would unwrap the future returned by divide () and raise    # The exception.    yield divide (1, 0)

Sometimes want to ' fire and forget ' a coroutine without waiting for its result. In this case it's recommended to use IOLoop.spawn_callback , which makes the responsible for the call IOLoop . If it fails, the would IOLoop log a stack trace:

Sometimes you may need to directly trigger a coroutine without waiting for its result, you need to ioloop.spawn_callback and let Ioiloop call it. If execution fails, ioloop logs the stack trace log. Because calling Coroutine directly doesn't get the result you want.

# The Ioloop would catch the exception and print a stack trace in # The logs. Note that this doesn ' t look like a normal call, since#  We pass the function object to being called by the I Oloop. Ioloop.current (). Spawn_callback (divide, 1, 0)

Finally, at the top level of a program, if the '. Ioloop ' isn't yet running, you can start IOLoop the, run the Coroutine, and then stop the with the IOLoop IOLoop.run_sync method . This is often used to start the main function of a batch-oriented program:

Eventually, at the top level of the program, if '. Ioloop ' is not running yet, you can use Ioloop to run coroutine, use the Ioloop.run_sync method to stop Ioloop. This is often used to run the main function. Simply put, if you want to execute a coroutine immediately, using Ioloop.spawn_callback, you want to perform a series of coroutine-composed programs, using Ioloop.run_sync.

# Run_sync () doesn ' t take arguments, so we must wrap the # Call in a lambda. Ioloop.current (). Run_sync (Lambda: Divide (1, 0))

Coroutine Patternsinteraction with callbacks

To interact with asynchronous code this uses callbacks instead Future of, wrap the call in a Task . This would add the callback argument for your and return a Future which you can yield:

Coroutine uses a Task to invoke a function when interacting with an asynchronous function that uses callback. The following sentence, add the callback argument is the principle of interpretation, it is better not to explain, directly said return a future more simple and clear.

@gen. Coroutine def call_task ():     # Note that there is no parens on some_function.    # This would be a translated by Task    into #    some_function (Other_args, callback=callback)    yield Gen. Task (Some_function, Other_args)

Calling blocking functions

The simplest-blocking function from a coroutine ThreadPoolExecutor was to use a, which returns this is Futures compatible W ITH coroutines:

The simplest way to invoke a blocking function is to use threadpoolexecutor, either IO bouding or CPU bouding, and Tornado is a single-threaded event loop that has a significant performance impact once a block is available. Threadpoolexecutor will return a future object

Thread_pool = Threadpoolexecutor (4) @gen. Coroutinedef  call_blocking ():    yield thread_pool.submit (Blocking_func, args)

Parallelism

The Coroutine decorator recognizes lists and dicts whose values Futures is, and waits for all of those in Futures parallel:

The Coroutine adorner will identify whether the values of lists and dicts are futures and will wait for all future objects to be executed in parallel.

@gen. CoroutinedefParallel_fetch (URL1, url2): Resp1, Resp2=yield[Http_client.fetch (URL1), Http_client.fetch (URL2)] @gen. CoroutinedefParallel_fetch_many (URLs): Responses=yield[Http_client.fetch (URL) forUrlinchURLs]#responses is a list of httpresponses in the same order@gen. Coroutinedefparallel_fetch_dict (URLs): Responses=yield{url:http_client.fetch (URL) forUrlinchURLs}#responses is a dict {url:httpresponse}

Interleaving

Sometimes it is useful to save a Future instead of yielding it immediately, so can start another operation before Waiti Ng

Sometimes you need to save a future instead of immediately yield it, so you can start another operation directly

@gen. Coroutine def Get (self):     = Self.fetch_next_chunk ()    while  True:        yield  fetch_future         ifis break        self.write (chunk)        = self.fetch_next_ Chunk ()        yield self.flush ()

Looping

Looping is tricky with coroutines since there is no-i-Python-on- yield every iteration of a for  or while loop and Capture the result of the yield. Instead, you'll need to separate the loop condition from accessing the results, as in this example from motor:

There is no method in Python that performs yield in each loop of a for or while, and gets the result of yield. To achieve a goal, you need to separate the loop conditions from the results obtained. I have no idea what this is all about. Does that mean, list and dict, for I in list, and then I in list, which drive the loop, and give you a coroutine, how to loop it to the end of execution, then wasted. In the example, the yield cursor.fetch_next is used as a cyclic condition and the data is obtained in doc = Cursor.next_object ().

Import= Motor . Motorclient (). Test@gen.coroutinedef  Loop_example (collection):    =  Db.collection.find ()    while (yield  cursor.fetch_next):        = Cursor.next_ Object ()

Running in the background

PeriodicCallbackIs isn't normally used with coroutines. Instead, a coroutine can contain a while True: loop and use tornado.gen.sleep :

To run the coroutine periodically, such as interval 60s, you need to use while True to loop with Tornado.gen.sleep.

Sometimes a more complicated loops may desirable. For example, the previous loop runs every 60+N seconds, where is the N running time of do_something() . To run exactly every seconds with the interleaving pattern from above:

Minute_loop, the interval between two do_something is 60s plus do_something run time

MINUTE_LOOP2, two do_something between the detection is 60s, of course, the premise is, do_something running time is less than 60s. If the do_something is running longer than 60s, it does not run every 60s. belongs to a proposition error. Then, you can infer that the two yield is executed in a synchronous order. If there is no @gen. Coroutine decoration, yield is asynchronous.

@gen. CoroutinedefMinute_loop (): whileTrue:yielddo_something ()yieldGen.sleep (60)#coroutines that loop forever is generally started with#spawn_callback ().ioloop.current (). Spawn_callback (Minute_loop)
@gen. Coroutinedefminute_loop2 (): whileTRUE:NXT= Gen.sleep (60)#Start the clock.        yieldDo_something ()#Run While the clock is ticking.        yieldNxt#Wait for the timer to run out.

Tornado Study Record II